Nov 03 20:00:21.662707 ip-10-0-142-5 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:00:22.017894 ip-10-0-142-5 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:00:22.017894 ip-10-0-142-5 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:00:22.017894 ip-10-0-142-5 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:00:22.017894 ip-10-0-142-5 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:00:22.022420 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022348 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022436 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022443 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022447 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022459 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022462 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022465 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022468 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022471 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022474 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022477 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022479 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:22.022480 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022484 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022488 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022492 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022494 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022497 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022500 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022502 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022505 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022508 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022516 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022519 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022522 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022524 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022527 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022529 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022532 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022535 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022537 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022540 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:22.022753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022542 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022545 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022547 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022550 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022552 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022554 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022557 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022559 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022562 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022564 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022567 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022569 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022573 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022576 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022579 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022583 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022585 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022588 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022591 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:22.023163 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022593 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022596 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022599 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022607 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022610 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022613 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022615 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022618 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022621 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022623 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022626 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022628 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022637 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022639 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022642 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022644 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022647 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022649 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022652 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:22.023797 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022654 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022657 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022660 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022662 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022664 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022667 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022669 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022672 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022674 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022678 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022680 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022683 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022686 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022688 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022691 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022693 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022696 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022704 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022708 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022711 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:22.024511 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022713 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022716 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022718 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022721 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022723 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022726 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022728 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022730 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022733 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022735 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022738 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022740 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022742 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022746 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022749 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022751 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.022754 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024506 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024523 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024529 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:22.025320 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024535 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024539 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024543 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024548 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024552 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024557 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024562 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024566 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024571 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024575 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024579 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024583 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024587 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024591 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024595 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024600 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024604 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024607 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024611 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024616 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:22.026130 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024620 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024623 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024628 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024631 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024636 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024640 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024644 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024648 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024653 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024659 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024663 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024668 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024673 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024677 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024681 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024685 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024689 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024693 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024697 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024701 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:22.026805 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024711 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024715 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024720 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024724 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024728 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024732 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024736 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024750 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024754 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024758 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024763 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024767 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024771 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024777 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024784 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024789 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024793 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024797 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024803 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:22.027266 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024807 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024811 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024816 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024820 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024824 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024828 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024832 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024839 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024845 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024850 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024855 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024860 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024864 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024867 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024872 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024876 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024881 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024885 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024889 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:22.028085 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024893 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024897 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024912 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024917 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024921 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024925 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024929 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024933 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024937 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024941 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024945 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024957 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024962 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024966 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024971 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024979 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024983 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024987 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024992 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:22.028935 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.024996 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025000 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025004 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025008 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025012 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025016 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025892 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025901 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025905 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025909 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025914 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025918 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025922 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025926 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025930 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025941 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025947 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025951 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025955 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:22.029733 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025960 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025964 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025968 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025972 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025976 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025981 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025985 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025990 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025994 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.025998 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026003 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026007 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026011 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026015 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026020 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026023 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026027 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026032 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026036 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026040 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:22.030278 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026044 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026048 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026052 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026056 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026060 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026064 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026068 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026072 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026076 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026088 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026092 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026097 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026101 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026105 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026109 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026113 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026117 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026121 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026125 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026129 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:22.030866 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026133 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026137 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026141 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026145 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026149 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026153 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026157 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026161 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026165 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026169 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026173 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026177 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026181 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026186 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026190 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026193 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026197 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026201 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026205 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:22.031314 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026209 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026213 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026217 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026229 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026234 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026253 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026258 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026262 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026266 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026270 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026275 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026280 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026284 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026288 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026292 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026297 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026301 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026305 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026309 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026313 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:22.031843 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026317 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026321 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026325 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026330 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026334 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026338 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026342 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026346 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026350 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026354 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026360 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026367 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026374 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.026381 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027278 2583 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027293 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027315 2583 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027323 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027330 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027335 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:00:22.032296 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027342 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027348 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027353 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027358 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027363 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027368 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027373 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027378 2583 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027382 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027387 2583 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027392 2583 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027396 2583 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027405 2583 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027410 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027414 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027420 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027440 2583 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027446 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027451 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027455 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027461 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027465 2583 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027472 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027477 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:00:22.032787 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027482 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027486 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027493 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027498 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027502 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027518 2583 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027523 2583 flags.go:64] FLAG: --enable-server="true" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027527 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027538 2583 flags.go:64] FLAG: --event-burst="100" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027544 2583 flags.go:64] FLAG: --event-qps="50" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027550 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027555 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027559 2583 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027566 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027571 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027576 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027581 2583 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027586 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027591 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027595 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027600 2583 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027604 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027609 2583 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027614 2583 flags.go:64] FLAG: --feature-gates="" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027620 2583 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:00:22.033340 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027625 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027630 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027634 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027639 2583 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027644 2583 flags.go:64] FLAG: --help="false" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027651 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027656 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027661 2583 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027666 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027671 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027677 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027682 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027686 2583 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027698 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027703 2583 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027708 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027713 2583 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027719 2583 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027724 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027729 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027734 2583 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027738 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027743 2583 flags.go:64] FLAG: --lock-file="" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027748 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:00:22.033922 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027752 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027757 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027767 2583 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027771 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027776 2583 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027780 2583 flags.go:64] FLAG: --logging-format="text" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027785 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027790 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027794 2583 flags.go:64] FLAG: --manifest-url="" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027798 2583 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027805 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027810 2583 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027816 2583 flags.go:64] FLAG: --max-pods="110" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027823 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027827 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027832 2583 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027836 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027841 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027846 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027850 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027860 2583 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027865 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027877 2583 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027882 2583 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027887 2583 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:00:22.034550 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027893 2583 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027898 2583 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027902 2583 flags.go:64] FLAG: --port="10250" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027907 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027912 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0581b9c4c2296ac53" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027917 2583 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027921 2583 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027926 2583 flags.go:64] FLAG: --register-node="true" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027930 2583 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027937 2583 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027941 2583 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027946 2583 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027950 2583 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027956 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027961 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027966 2583 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027970 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027975 2583 flags.go:64] FLAG: --runonce="false" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027979 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027985 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027992 2583 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.027996 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028001 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028005 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028011 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028015 2583 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:00:22.035128 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028020 2583 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028024 2583 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028029 2583 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028033 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028046 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028051 2583 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028058 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028066 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028070 2583 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028075 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028084 2583 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028088 2583 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028092 2583 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028097 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028101 2583 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028106 2583 flags.go:64] FLAG: --v="2" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028112 2583 flags.go:64] FLAG: --version="false" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028119 2583 flags.go:64] FLAG: --vmodule="" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028125 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028130 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028348 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028356 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028362 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028367 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:22.035753 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028372 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028376 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028383 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028390 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028394 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028399 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028404 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028408 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028412 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028416 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028436 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028441 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028445 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028459 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028467 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028471 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028475 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028479 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028483 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028488 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:22.036286 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028491 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028495 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028500 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028504 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028508 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028512 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028516 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028520 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028524 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028529 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028533 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028537 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028541 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028545 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028551 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028556 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028560 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028564 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028570 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:22.036778 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028577 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028582 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028587 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028591 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028595 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028599 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028604 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028617 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028622 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028626 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028631 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028636 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028642 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028647 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028652 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028657 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028661 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028665 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028669 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:22.037242 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028673 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028677 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028681 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028685 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028689 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028694 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028698 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028703 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028714 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028719 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028723 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028727 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028732 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028735 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028739 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028743 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028748 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028752 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028756 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028760 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:22.037695 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028766 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028777 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028782 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028786 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028791 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028801 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028806 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028810 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028814 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028818 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028822 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028826 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028831 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028835 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028839 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028843 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028848 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028852 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028856 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:22.038184 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028860 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028866 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028871 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028875 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028879 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.028925 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.028993 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029001 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029006 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029010 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029015 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029019 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029023 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029027 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029032 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029043 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:22.038634 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029047 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029051 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029055 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029060 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029064 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029068 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029072 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029076 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029080 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029084 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029088 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029092 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029097 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029101 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029105 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029109 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029113 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029119 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029126 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:22.038999 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029131 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029135 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029140 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029144 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029149 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029153 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029157 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029161 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029165 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029169 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029173 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029178 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029184 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029197 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029202 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029206 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029210 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029213 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029217 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:22.039440 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029222 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029226 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029230 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029234 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029238 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029242 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029246 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029250 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029254 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029258 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029263 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029269 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029273 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029277 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029281 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029285 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029289 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029293 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029297 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029301 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:22.039955 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029305 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029310 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029314 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029318 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029322 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029327 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029332 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029343 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029348 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029352 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029356 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029360 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029364 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029369 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029372 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029376 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029380 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029384 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029388 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:22.040410 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029392 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029396 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029400 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029404 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029410 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029415 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029419 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029440 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029445 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029449 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029452 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029456 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029470 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029475 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029479 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029483 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029487 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029491 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.029495 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.036856 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:00:22.040863 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.036870 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.036887 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.036892 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036951 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036956 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036960 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036962 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036965 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036968 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036970 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036973 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036975 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036978 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036980 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036983 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036985 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036988 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036990 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036992 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:22.041310 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036995 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.036998 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037000 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037003 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037005 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037008 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037010 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037013 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037015 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037018 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037020 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037023 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037025 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037029 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037032 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037035 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037038 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037040 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037043 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037045 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:22.041754 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037047 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037050 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037052 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037055 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037057 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037060 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037062 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037064 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037067 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037069 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037072 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037074 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037076 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037079 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037082 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037084 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037087 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037089 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037092 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:22.042202 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037094 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037096 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037099 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037102 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037104 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037107 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037110 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037112 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037115 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037117 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037120 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037122 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037125 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037127 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037129 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037132 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037134 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037137 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037139 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:22.042651 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037142 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037144 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037147 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037149 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037152 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037154 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037156 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037159 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037161 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037164 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037166 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037169 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037171 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037174 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037176 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037180 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037184 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037187 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037191 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:22.043116 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037195 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037198 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037200 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037204 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037206 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037209 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037211 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037214 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037216 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037219 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037221 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037223 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037226 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.037253 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037401 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037406 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:22.043557 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037409 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037412 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037414 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037417 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037420 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037437 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037443 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037446 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037448 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037451 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037453 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037456 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037458 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037461 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037464 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037466 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037469 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037471 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037474 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037477 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:22.043905 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037480 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037483 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037485 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037488 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037490 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037493 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037495 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037498 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037500 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037503 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037505 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037507 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037511 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037515 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037517 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037519 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037522 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037524 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037527 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:22.044344 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037529 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037532 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037534 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037537 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037539 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037541 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037544 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037546 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037549 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037551 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037554 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037556 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037559 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037561 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037564 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037567 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037569 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037572 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037574 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037576 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:22.044884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037579 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037581 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037584 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037586 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037588 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037591 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037593 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037596 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037598 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037601 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037603 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037606 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037608 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037611 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037613 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037616 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037618 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037621 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037623 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037626 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:22.045376 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037629 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037631 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037633 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037636 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037638 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037641 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037643 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037646 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037648 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037651 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037653 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037656 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037660 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037663 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037665 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037668 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037670 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037673 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037675 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:22.045927 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037678 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037680 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037683 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037685 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037688 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037690 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.037712 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037748 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037753 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037755 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037758 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037761 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037764 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037770 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037773 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:00:22.046369 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037776 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037778 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037781 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037784 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037786 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037789 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037792 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037794 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037797 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037799 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037802 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037804 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037806 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037809 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037813 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037816 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037818 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037821 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037823 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037826 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:00:22.046723 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037828 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037831 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037833 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037836 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037838 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037841 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037843 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037846 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037848 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037851 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037854 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037857 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037860 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037862 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037865 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037867 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037871 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037875 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037877 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:00:22.047169 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037881 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037883 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037886 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037889 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037892 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037894 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037897 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037900 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037903 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037906 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037908 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037911 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037913 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037915 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037918 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037920 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037923 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037925 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037928 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:00:22.047609 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037930 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037933 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037935 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037938 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037941 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037944 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037946 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037949 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037951 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037954 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037956 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037959 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037961 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037964 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037967 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037970 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037973 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037975 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037977 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037980 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:00:22.048033 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037984 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037987 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037989 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037992 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037994 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037997 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.037999 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038002 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038004 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038007 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038009 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038011 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038014 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038016 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038019 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038021 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038028 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038032 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038035 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:00:22.048507 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:22.038037 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:00:22.048974 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.038083 2583 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:00:22.048974 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.039910 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 20:00:22.048974 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.040958 2583 server.go:1014] "Starting client certificate rotation" Nov 03 20:00:22.048974 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.041054 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:00:22.048974 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.041096 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:00:22.048974 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.043509 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:00:22.048974 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.044577 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:00:22.053620 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.053605 2583 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:00:22.057463 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.057446 2583 log.go:25] "Validated CRI v1 image API" Nov 03 20:00:22.058193 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.058171 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:00:22.060289 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.060270 2583 fs.go:135] Filesystem UUIDs: map[4f7bb23f-23d6-4ee9-adad-5d1a0717478f:/dev/nvme0n1p4 5b4d2772-1258-45af-b013-31ec7ffdea05:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 20:00:22.060361 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.060288 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 20:00:22.065390 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.065279 2583 manager.go:217] Machine: {Timestamp:2026-11-03 20:00:22.064204344 +0000 UTC m=+0.298205590 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199898 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec283fabaf69111b6b6d2761d33fe7b6 SystemUUID:ec283fab-af69-111b-6b6d-2761d33fe7b6 BootID:5c2453d4-616a-4396-8552-391b411092a0 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:24:85:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:24:85:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:9a:56:b0:41:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:00:22.065794 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.065784 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:00:22.065870 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.065859 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:00:22.068473 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.068443 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:00:22.068622 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.068474 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-5.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:00:22.068622 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.068623 2583 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:00:22.068751 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.068631 2583 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:00:22.068751 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.068647 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:00:22.069450 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.069439 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:00:22.070074 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.070064 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:00:22.073493 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.073480 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:00:22.073611 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.073601 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:00:22.073640 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.073635 2583 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:00:22.073675 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.073648 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:00:22.073675 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.073658 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:00:22.073675 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.073668 2583 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:00:22.073675 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.073675 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:00:22.073963 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.073951 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:00:22.075441 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.075403 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:00:22.076070 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.076057 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:00:22.080238 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.080221 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:00:22.081116 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081101 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:00:22.081188 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081140 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:00:22.081325 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081312 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:00:22.081382 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081331 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:00:22.081382 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081341 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:00:22.081382 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081350 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:00:22.081382 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081358 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:00:22.081382 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081367 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:00:22.081831 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081821 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:00:22.081881 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081836 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:00:22.081881 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081847 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:00:22.081881 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081868 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:00:22.082061 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081885 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:00:22.082061 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.081900 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:00:22.082536 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.082525 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:00:22.082584 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.082540 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:00:22.086020 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.086004 2583 server.go:1263] "Started kubelet" Nov 03 20:00:22.087013 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.086081 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:00:22.087013 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.086124 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:00:22.087013 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.086252 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:00:22.086985 ip-10-0-142-5 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:00:22.087415 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.087136 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:00:22.087482 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.087443 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-5.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 20:00:22.088000 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.087988 2583 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:00:22.091061 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.091031 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 20:00:22.094838 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.094819 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:00:22.095801 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.095787 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:00:22.095960 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.095914 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.096324 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096306 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:00:22.096324 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096324 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:00:22.096476 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096304 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:00:22.096476 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096456 2583 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:00:22.096476 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096464 2583 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:00:22.096647 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096634 2583 factory.go:55] Registering systemd factory Nov 03 20:00:22.096691 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096648 2583 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:00:22.096878 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096860 2583 factory.go:153] Registering CRI-O factory Nov 03 20:00:22.096878 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096878 2583 factory.go:223] Registration of the crio container factory successfully Nov 03 20:00:22.097028 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096979 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:00:22.097028 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.096997 2583 factory.go:103] Registering Raw factory Nov 03 20:00:22.097028 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.097012 2583 manager.go:1199] Started watching for new ooms in manager Nov 03 20:00:22.097747 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.097715 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 20:00:22.097903 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.097889 2583 manager.go:319] Starting recovery of all containers Nov 03 20:00:22.098062 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.098042 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-5.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 20:00:22.098238 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.097456 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-5.us-east-2.compute.internal.18e4a1689141cecf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-5.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-5.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:22.085979855 +0000 UTC m=+0.319981101,LastTimestamp:2026-11-03 20:00:22.085979855 +0000 UTC m=+0.319981101,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-5.us-east-2.compute.internal,}" Nov 03 20:00:22.108485 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.107970 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h9zpr" Nov 03 20:00:22.108485 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.108003 2583 manager.go:324] Recovery completed Nov 03 20:00:22.108485 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.108329 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-5.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 20:00:22.112468 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.112412 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:22.114476 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.114461 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:22.114584 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.114489 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:22.114584 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.114502 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:22.115067 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.115054 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:22.115162 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.115054 2583 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:00:22.115162 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.115104 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:00:22.115162 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.115120 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:00:22.115597 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.115560 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 20:00:22.115873 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.115852 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:22.115948 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.115889 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:22.115948 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.115904 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:22.117133 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.117118 2583 policy_none.go:50] "Start" Nov 03 20:00:22.117202 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.117135 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:00:22.117202 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.117148 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:00:22.119026 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.119012 2583 policy_none.go:44] "Start" Nov 03 20:00:22.121547 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.121530 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h9zpr" Nov 03 20:00:22.123666 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.123597 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-5.us-east-2.compute.internal.18e4a16892f49cf4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-5.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-5.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-5.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 20:00:22.114475252 +0000 UTC m=+0.348476499,LastTimestamp:2026-11-03 20:00:22.114475252 +0000 UTC m=+0.348476499,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-5.us-east-2.compute.internal,}" Nov 03 20:00:22.153952 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.153926 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:00:22.157630 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.157618 2583 manager.go:341] "Starting Device Plugin manager" Nov 03 20:00:22.157711 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.157678 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:00:22.157711 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.157687 2583 server.go:87] "Starting device plugin registration server" Nov 03 20:00:22.157897 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.157886 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:00:22.157948 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.157898 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:00:22.158088 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.158072 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:00:22.158200 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.158144 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:00:22.158200 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.158152 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:00:22.158664 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.158644 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:00:22.158751 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.158690 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.258657 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.258620 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:22.260574 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.259651 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:22.260574 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.259688 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:22.260574 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.259698 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:22.260574 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.259725 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.267987 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.267926 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.267987 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.267949 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-5.us-east-2.compute.internal\": node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.286673 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.286538 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:00:22.287823 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.287802 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:00:22.287823 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.287825 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:00:22.287970 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.287848 2583 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:00:22.287970 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.287885 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 20:00:22.289220 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.289199 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.295392 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.295373 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:22.388741 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.388710 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal"] Nov 03 20:00:22.388881 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.388778 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:22.389440 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.389412 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.389756 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.389742 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:22.389810 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.389772 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:22.389810 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.389788 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:22.392116 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.392103 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:22.392265 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.392252 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.392304 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.392281 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:22.392843 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.392826 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:22.392843 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.392836 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:22.392959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.392855 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:22.392959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.392861 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:22.392959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.392868 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:22.392959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.392874 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:22.394985 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.394970 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.395052 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.394997 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:22.395818 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.395803 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:22.395868 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.395829 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:22.395868 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.395862 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:22.397986 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.397963 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23259ea2648a683e23b5758ee6ae68e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"23259ea2648a683e23b5758ee6ae68e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.398068 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.397994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.398068 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.398023 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.425906 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.425888 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" node="ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.432404 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.432388 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" node="ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.489499 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.489476 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.498797 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.498781 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23259ea2648a683e23b5758ee6ae68e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"23259ea2648a683e23b5758ee6ae68e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.498876 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.498803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.498876 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.498828 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.498972 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.498878 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23259ea2648a683e23b5758ee6ae68e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"23259ea2648a683e23b5758ee6ae68e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.498972 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.498873 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.498972 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.498908 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.590417 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.590343 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.690892 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.690854 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.728291 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.728271 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.734998 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:22.734979 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:22.791772 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.791748 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.892284 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.892226 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:22.992792 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:22.992756 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:23.041191 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:23.041172 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 20:00:23.041587 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:23.041290 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:00:23.093062 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:23.093041 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:23.125963 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:23.125909 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:55:23 +0000 UTC" deadline="2028-06-14 16:50:18.44697195 +0000 UTC" Nov 03 20:00:23.125963 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:23.125960 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14132h49m55.321013989s" Nov 03 20:00:23.193187 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:23.193141 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:24.649696 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:24.649678 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:00:24.650990 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:24.650972 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:00:24.651079 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:24.651003 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:00:24.651079 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:24.651016 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:00:24.662243 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:24.662219 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:24.721357 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:24.721334 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:24.766350 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:24.766322 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62881be2bc7cb00291f2b39128d27e0c.slice/crio-46d08c0c52558027466795f3a7a09ab2a28e10d941d2fa1bff4dbc7cff288ea8 WatchSource:0}: Error finding container 46d08c0c52558027466795f3a7a09ab2a28e10d941d2fa1bff4dbc7cff288ea8: Status 404 returned error can't find the container with id 46d08c0c52558027466795f3a7a09ab2a28e10d941d2fa1bff4dbc7cff288ea8 Nov 03 20:00:24.766770 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:24.766753 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23259ea2648a683e23b5758ee6ae68e7.slice/crio-bb03c65f0709ceac8b7ab5131b0c4df7fb28c2cc6c669d49b14aed783cfcb2b6 WatchSource:0}: Error finding container bb03c65f0709ceac8b7ab5131b0c4df7fb28c2cc6c669d49b14aed783cfcb2b6: Status 404 returned error can't find the container with id bb03c65f0709ceac8b7ab5131b0c4df7fb28c2cc6c669d49b14aed783cfcb2b6 Nov 03 20:00:24.771947 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:24.771902 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:00:24.822090 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:24.822051 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:24.922619 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:24.922562 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-5.us-east-2.compute.internal\" not found" Nov 03 20:00:24.982248 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:24.982227 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:24.996192 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:24.996168 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:25.024039 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.024022 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:25.036078 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.036059 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:00:25.037111 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.037098 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:00:25.046307 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.046291 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:00:25.502970 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.502940 2583 apiserver.go:52] "Watching apiserver" Nov 03 20:00:25.512014 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.511989 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:00:25.512321 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.512289 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-56d9g","openshift-network-diagnostics/network-check-target-4xx8n","kube-system/global-pull-secret-syncer-bhl7x","openshift-ovn-kubernetes/ovnkube-node-txw5p","kube-system/konnectivity-agent-8fpvs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr","openshift-cluster-node-tuning-operator/tuned-qc2tr","openshift-image-registry/node-ca-5h7vn","openshift-multus/multus-additional-cni-plugins-s7lv9","openshift-multus/network-metrics-daemon-xf24w","openshift-network-operator/iptables-alerter-tffb6","kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal","openshift-dns/node-resolver-wszxg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal"] Nov 03 20:00:25.514567 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.514544 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.516788 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.516768 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.521346 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.519655 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:25.521346 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.519836 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:00:25.521346 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.520565 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:25.521346 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.520583 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s55dl\"" Nov 03 20:00:25.521346 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.520389 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:25.521346 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.521129 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:00:25.521346 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.521193 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:00:25.521346 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.521289 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8fc4f\"" Nov 03 20:00:25.522746 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.522724 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.523243 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.523228 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 20:00:25.525220 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.525117 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:25.527419 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.527399 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.529698 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.529681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.532119 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.532101 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.535073 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.534702 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:25.535073 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.534762 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:25.536282 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536262 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-config\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.536376 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536288 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-script-lib\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.536376 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536304 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7dj\" (UniqueName: \"kubernetes.io/projected/43579876-2c2b-4730-bd82-717f9b116bab-kube-api-access-hj7dj\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.536376 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536318 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:25.536376 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.536376 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536350 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-netd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.536376 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-systemd-units\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536381 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-modprobe-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536397 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-kubernetes\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536412 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-sys\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536427 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-etc-tuned\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536464 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fbdb17a3-3ead-4c90-9922-31fbae907537-hosts-file\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536485 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-kubelet\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536533 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-log-socket\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536578 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysconfig\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-systemd\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.536714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-lib-modules\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-slash\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536755 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-netns\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536789 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-var-lib-kubelet\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536816 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-var-lib-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536838 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-etc-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536866 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af6745a-681c-44b6-861c-310df033cfe9-ovn-node-metrics-cert\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536875 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536888 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8w7\" (UniqueName: \"kubernetes.io/projected/9af6745a-681c-44b6-861c-310df033cfe9-kube-api-access-7g8w7\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536910 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536931 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-run\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536974 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-host\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.536994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdb17a3-3ead-4c90-9922-31fbae907537-tmp-dir\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-systemd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-node-log\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/042f2f33-8a62-4464-bdd8-1a54ed18272d-agent-certs\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537114 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flsn\" (UniqueName: \"kubernetes.io/projected/fbdb17a3-3ead-4c90-9922-31fbae907537-kube-api-access-9flsn\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.537222 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537134 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-kubelet-config\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537176 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-ovn\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-env-overrides\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537244 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-dbus\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/042f2f33-8a62-4464-bdd8-1a54ed18272d-konnectivity-ca\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537280 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537287 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-conf\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537308 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-tmp\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537329 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537350 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-bin\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.537370 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:25.537959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.537389 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.540502 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.539721 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.542055 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.542027 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.559354 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.559336 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8vpjw" Nov 03 20:00:25.573645 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.573627 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8vpjw" Nov 03 20:00:25.618734 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.618708 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:00:25.619349 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.619199 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:00:25.619349 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.619275 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:00:25.619525 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.619466 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:00:25.619616 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.619597 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:00:25.619715 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.619693 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:00:25.619815 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.619799 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lxhjg\"" Nov 03 20:00:25.619983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.619966 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:00:25.620132 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620117 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vbl6p\"" Nov 03 20:00:25.620199 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620161 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:00:25.620232 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620211 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:00:25.620291 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620280 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:00:25.620336 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620328 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kmt87\"" Nov 03 20:00:25.620378 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620357 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:00:25.620410 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620377 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:00:25.620597 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620578 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:00:25.620668 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620654 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:00:25.620760 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620739 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:00:25.620827 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620791 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:00:25.620827 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620803 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:00:25.620924 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620827 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:00:25.620924 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620836 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:00:25.620924 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.620878 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:00:25.622185 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.622160 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vlwgf\"" Nov 03 20:00:25.622268 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.622214 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnml9\"" Nov 03 20:00:25.622268 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.622231 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-whl79\"" Nov 03 20:00:25.622268 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.622243 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:00:25.622379 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.622277 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:00:25.622379 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.622286 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zfqpt\"" Nov 03 20:00:25.622379 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.622168 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:00:25.626161 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.626145 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:00:25.638407 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638388 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-etc-tuned\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.638519 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fbdb17a3-3ead-4c90-9922-31fbae907537-hosts-file\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.638519 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysconfig\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.638519 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-systemd\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.638519 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638516 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-systemd\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638524 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-lib-modules\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638556 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-netns\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638531 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysconfig\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638584 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-socket-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638529 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fbdb17a3-3ead-4c90-9922-31fbae907537-hosts-file\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638607 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-netns\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638606 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-daemon-config\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638658 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-lib-modules\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638666 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-var-lib-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638706 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-var-lib-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.638723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638710 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8w7\" (UniqueName: \"kubernetes.io/projected/9af6745a-681c-44b6-861c-310df033cfe9-kube-api-access-7g8w7\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638754 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638796 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-iptables-alerter-script\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638826 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638861 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-device-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638875 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-etc-selinux\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638888 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks7jx\" (UniqueName: \"kubernetes.io/projected/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-kube-api-access-ks7jx\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638936 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-run\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-systemd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638965 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-node-log\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.638986 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flsn\" (UniqueName: \"kubernetes.io/projected/fbdb17a3-3ead-4c90-9922-31fbae907537-kube-api-access-9flsn\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639059 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-kubelet-config\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639098 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-cnibin\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639113 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639130 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-registration-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-cnibin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.639226 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639160 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-dbus\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639175 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/042f2f33-8a62-4464-bdd8-1a54ed18272d-konnectivity-ca\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639202 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-node-log\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639247 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsj6v\" (UniqueName: \"kubernetes.io/projected/d938e951-475c-47d6-8d73-f17da86400ad-kube-api-access-dsj6v\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-kubelet\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639286 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-conf-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639303 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-conf\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639318 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-bin\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-script-lib\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639368 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-systemd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639380 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639397 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7dj\" (UniqueName: \"kubernetes.io/projected/43579876-2c2b-4730-bd82-717f9b116bab-kube-api-access-hj7dj\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639517 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-run\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639556 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-kubelet-config\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639580 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639621 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-system-cni-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.639935 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639651 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-systemd-units\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639685 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-modprobe-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639712 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-kubernetes\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639652 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-conf\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639765 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-dbus\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639773 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-sys\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639791 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-systemd-units\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-kubelet\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639825 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-log-socket\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639848 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-kubernetes\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639850 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/baba1acd-b516-4149-a337-d56b9e74b36e-serviceca\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz86h\" (UniqueName: \"kubernetes.io/projected/baba1acd-b516-4149-a337-d56b9e74b36e-kube-api-access-dz86h\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639922 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-modprobe-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-slash\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639942 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-bin\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639950 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/042f2f33-8a62-4464-bdd8-1a54ed18272d-konnectivity-ca\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639961 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-cni-binary-copy\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.640566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639978 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-script-lib\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.639995 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-sys\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baba1acd-b516-4149-a337-d56b9e74b36e-host\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640032 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-kubelet\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640041 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-slash\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640049 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-var-lib-kubelet\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640071 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-log-socket\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640083 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-etc-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640093 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-var-lib-kubelet\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640137 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af6745a-681c-44b6-861c-310df033cfe9-ovn-node-metrics-cert\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640145 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-etc-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640181 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-host-slash\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640431 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-system-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640470 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-etc-kubernetes\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640514 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640543 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-host\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdb17a3-3ead-4c90-9922-31fbae907537-tmp-dir\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.641343 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640706 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-host\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640706 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/042f2f33-8a62-4464-bdd8-1a54ed18272d-agent-certs\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640740 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640755 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640780 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-sys-fs\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-multus-certs\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640839 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640876 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-ovn\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-env-overrides\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640931 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640941 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-ovn\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.640964 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdb17a3-3ead-4c90-9922-31fbae907537-tmp-dir\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641004 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxr52\" (UniqueName: \"kubernetes.io/projected/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kube-api-access-rxr52\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-socket-dir-parent\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641104 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-netns\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641145 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvms\" (UniqueName: \"kubernetes.io/projected/a3f23518-97e0-4f3f-b939-d76052dc1339-kube-api-access-sjvms\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-k8s-cni-cncf-io\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.642150 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641191 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-tmp\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641215 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-config\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641280 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gb7w\" (UniqueName: \"kubernetes.io/projected/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-kube-api-access-7gb7w\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641306 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-multus\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641313 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-env-overrides\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641329 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-hostroot\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641391 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641417 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-netd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641492 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-bin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.641586 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.641754 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.641816 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret podName:a9998949-10c8-4511-857c-dfad4001ab59 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:26.14179856 +0000 UTC m=+2.947779130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret") pod "global-pull-secret-syncer-bhl7x" (UID: "a9998949-10c8-4511-857c-dfad4001ab59") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.642017 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-config\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.642045 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-netd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.642181 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.642975 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.642404 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-etc-tuned\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.643622 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.643496 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af6745a-681c-44b6-861c-310df033cfe9-ovn-node-metrics-cert\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.643622 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.643614 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-tmp\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.643998 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.643975 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/042f2f33-8a62-4464-bdd8-1a54ed18272d-agent-certs\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:25.651005 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.650981 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8w7\" (UniqueName: \"kubernetes.io/projected/9af6745a-681c-44b6-861c-310df033cfe9-kube-api-access-7g8w7\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.651963 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.651943 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flsn\" (UniqueName: \"kubernetes.io/projected/fbdb17a3-3ead-4c90-9922-31fbae907537-kube-api-access-9flsn\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.653241 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.653222 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7dj\" (UniqueName: \"kubernetes.io/projected/43579876-2c2b-4730-bd82-717f9b116bab-kube-api-access-hj7dj\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.720828 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.720785 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"62881be2bc7cb00291f2b39128d27e0c","Type":"ContainerStarted","Data":"46d08c0c52558027466795f3a7a09ab2a28e10d941d2fa1bff4dbc7cff288ea8"} Nov 03 20:00:25.721756 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.721726 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"23259ea2648a683e23b5758ee6ae68e7","Type":"ContainerStarted","Data":"bb03c65f0709ceac8b7ab5131b0c4df7fb28c2cc6c669d49b14aed783cfcb2b6"} Nov 03 20:00:25.741971 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.741940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-conf-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.742081 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.741982 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.742081 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-system-cni-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.742081 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/baba1acd-b516-4149-a337-d56b9e74b36e-serviceca\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.742081 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742054 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz86h\" (UniqueName: \"kubernetes.io/projected/baba1acd-b516-4149-a337-d56b9e74b36e-kube-api-access-dz86h\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.742081 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742061 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-conf-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.742081 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742074 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-cni-binary-copy\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.742365 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742095 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:25.742365 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742122 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-system-cni-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.742365 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.742183 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:25.742365 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.742244 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs podName:dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:26.242226403 +0000 UTC m=+3.048206961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs") pod "network-metrics-daemon-xf24w" (UID: "dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:25.742365 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baba1acd-b516-4149-a337-d56b9e74b36e-host\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-host-slash\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742428 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-host-slash\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742432 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baba1acd-b516-4149-a337-d56b9e74b36e-host\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742478 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-system-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-etc-kubernetes\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742531 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742556 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-sys-fs\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-multus-certs\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742594 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-etc-kubernetes\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742599 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742599 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742605 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-cni-binary-copy\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.742649 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742650 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742664 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-sys-fs\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742672 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-system-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742676 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxr52\" (UniqueName: \"kubernetes.io/projected/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kube-api-access-rxr52\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742719 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-multus-certs\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742746 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-socket-dir-parent\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742769 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-netns\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742793 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvms\" (UniqueName: \"kubernetes.io/projected/a3f23518-97e0-4f3f-b939-d76052dc1339-kube-api-access-sjvms\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742815 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-k8s-cni-cncf-io\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742840 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gb7w\" (UniqueName: \"kubernetes.io/projected/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-kube-api-access-7gb7w\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-multus\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742881 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-hostroot\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-bin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742946 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-socket-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-daemon-config\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.742992 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-iptables-alerter-script\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743049 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-socket-dir-parent\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743229 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743084 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743094 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743109 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-device-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743135 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-etc-selinux\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743143 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-hostroot\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743158 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743172 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-k8s-cni-cncf-io\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks7jx\" (UniqueName: \"kubernetes.io/projected/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-kube-api-access-ks7jx\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743213 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-cnibin\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743237 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743236 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743295 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-netns\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743318 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-socket-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743340 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743354 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-registration-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743386 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-cnibin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743389 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-multus\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.743983 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743392 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-device-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743461 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-cnibin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743488 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-iptables-alerter-script\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743502 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-bin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743527 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-etc-selinux\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743545 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-registration-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743581 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-cnibin\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743604 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsj6v\" (UniqueName: \"kubernetes.io/projected/d938e951-475c-47d6-8d73-f17da86400ad-kube-api-access-dsj6v\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743635 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743635 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-kubelet\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743668 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-kubelet\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743684 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.743856 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-daemon-config\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.744643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.744586 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/baba1acd-b516-4149-a337-d56b9e74b36e-serviceca\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.755872 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.755822 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:25.755872 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.755846 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:25.755872 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.755859 2583 projected.go:196] Error preparing data for projected volume kube-api-access-wsfgx for pod openshift-network-diagnostics/network-check-target-4xx8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:25.756072 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:25.755924 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx podName:a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b nodeName:}" failed. No retries permitted until 2026-11-03 20:00:26.255907347 +0000 UTC m=+3.061887915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfgx" (UniqueName: "kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx") pod "network-check-target-4xx8n" (UID: "a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:25.757161 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.757120 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz86h\" (UniqueName: \"kubernetes.io/projected/baba1acd-b516-4149-a337-d56b9e74b36e-kube-api-access-dz86h\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.758756 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.758738 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gb7w\" (UniqueName: \"kubernetes.io/projected/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-kube-api-access-7gb7w\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.759606 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.759589 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks7jx\" (UniqueName: \"kubernetes.io/projected/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-kube-api-access-ks7jx\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:25.761005 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.760985 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsj6v\" (UniqueName: \"kubernetes.io/projected/d938e951-475c-47d6-8d73-f17da86400ad-kube-api-access-dsj6v\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.761087 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.761019 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvms\" (UniqueName: \"kubernetes.io/projected/a3f23518-97e0-4f3f-b939-d76052dc1339-kube-api-access-sjvms\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.762250 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.762230 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxr52\" (UniqueName: \"kubernetes.io/projected/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kube-api-access-rxr52\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.831375 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.831347 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:00:25.838667 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.838644 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wszxg" Nov 03 20:00:25.842186 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:25.842166 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43579876_2c2b_4730_bd82_717f9b116bab.slice/crio-fedd28b690458e403ed90cc0c8fc02c863c55e5a25868849590c9dda395353c8 WatchSource:0}: Error finding container fedd28b690458e403ed90cc0c8fc02c863c55e5a25868849590c9dda395353c8: Status 404 returned error can't find the container with id fedd28b690458e403ed90cc0c8fc02c863c55e5a25868849590c9dda395353c8 Nov 03 20:00:25.847175 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:25.847150 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdb17a3_3ead_4c90_9922_31fbae907537.slice/crio-88e618455f8f22e472094ce96e7ddd4ffb1c16231f006da80f767e0977d456fe WatchSource:0}: Error finding container 88e618455f8f22e472094ce96e7ddd4ffb1c16231f006da80f767e0977d456fe: Status 404 returned error can't find the container with id 88e618455f8f22e472094ce96e7ddd4ffb1c16231f006da80f767e0977d456fe Nov 03 20:00:25.859407 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.859389 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:25.862583 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.862558 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:00:25.868838 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.868820 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:00:25.869189 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:25.869169 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042f2f33_8a62_4464_bdd8_1a54ed18272d.slice/crio-d37bd4fe64f280a4e42d76022d6f13a2367dabd5d16c77277c3161b8ceaf480a WatchSource:0}: Error finding container d37bd4fe64f280a4e42d76022d6f13a2367dabd5d16c77277c3161b8ceaf480a: Status 404 returned error can't find the container with id d37bd4fe64f280a4e42d76022d6f13a2367dabd5d16c77277c3161b8ceaf480a Nov 03 20:00:25.872218 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:25.872197 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af6745a_681c_44b6_861c_310df033cfe9.slice/crio-f0cba8e705823b06d02bad90ef7f7806afaa9ef45e6d46342e1ceaea064538fe WatchSource:0}: Error finding container f0cba8e705823b06d02bad90ef7f7806afaa9ef45e6d46342e1ceaea064538fe: Status 404 returned error can't find the container with id f0cba8e705823b06d02bad90ef7f7806afaa9ef45e6d46342e1ceaea064538fe Nov 03 20:00:25.878131 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:25.878108 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcea6c8_6250_4a3e_9f1e_d4ea3c02c52f.slice/crio-af3faa9407b7346ff5c2f1886300b5fdb7853496a98153bfc21c30fa280adda6 WatchSource:0}: Error finding container af3faa9407b7346ff5c2f1886300b5fdb7853496a98153bfc21c30fa280adda6: Status 404 returned error can't find the container with id af3faa9407b7346ff5c2f1886300b5fdb7853496a98153bfc21c30fa280adda6 Nov 03 20:00:25.879491 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.879475 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-56d9g" Nov 03 20:00:25.886544 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.886527 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:00:25.888881 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.888863 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:00:25.889407 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:25.889388 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f23518_97e0_4f3f_b939_d76052dc1339.slice/crio-622e2d08eee35041ae23385941f7c82d5a4d4a7847d17ff0604944ba4cdf0258 WatchSource:0}: Error finding container 622e2d08eee35041ae23385941f7c82d5a4d4a7847d17ff0604944ba4cdf0258: Status 404 returned error can't find the container with id 622e2d08eee35041ae23385941f7c82d5a4d4a7847d17ff0604944ba4cdf0258 Nov 03 20:00:25.891265 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:25.891248 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:00:25.896120 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:25.896007 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd938e951_475c_47d6_8d73_f17da86400ad.slice/crio-f9c98cbda157f34909384446fce81fe2770c3486fb529684caadcc23623a97f4 WatchSource:0}: Error finding container f9c98cbda157f34909384446fce81fe2770c3486fb529684caadcc23623a97f4: Status 404 returned error can't find the container with id f9c98cbda157f34909384446fce81fe2770c3486fb529684caadcc23623a97f4 Nov 03 20:00:25.903588 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:00:25.903568 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaba1acd_b516_4149_a337_d56b9e74b36e.slice/crio-24033f77e98e2b98387bbff636ca30f4fbcb5c270f4e650dfba84d475b2b87e9 WatchSource:0}: Error finding container 24033f77e98e2b98387bbff636ca30f4fbcb5c270f4e650dfba84d475b2b87e9: Status 404 returned error can't find the container with id 24033f77e98e2b98387bbff636ca30f4fbcb5c270f4e650dfba84d475b2b87e9 Nov 03 20:00:26.043638 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.043571 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:00:26.146857 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.146830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:26.146994 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:26.146976 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:26.147047 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:26.147035 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret podName:a9998949-10c8-4511-857c-dfad4001ab59 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:27.147022266 +0000 UTC m=+3.953002818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret") pod "global-pull-secret-syncer-bhl7x" (UID: "a9998949-10c8-4511-857c-dfad4001ab59") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:26.247903 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.247873 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:26.248031 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:26.248018 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:26.248079 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:26.248070 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs podName:dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:27.248057956 +0000 UTC m=+4.054038512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs") pod "network-metrics-daemon-xf24w" (UID: "dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:26.348937 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.348900 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:26.349082 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:26.349070 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:26.349145 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:26.349088 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:26.349145 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:26.349099 2583 projected.go:196] Error preparing data for projected volume kube-api-access-wsfgx for pod openshift-network-diagnostics/network-check-target-4xx8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:26.349244 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:26.349161 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx podName:a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b nodeName:}" failed. No retries permitted until 2026-11-03 20:00:27.34914325 +0000 UTC m=+4.155123804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfgx" (UniqueName: "kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx") pod "network-check-target-4xx8n" (UID: "a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:26.576953 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.576773 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:55:25 +0000 UTC" deadline="2028-05-19 01:15:05.472882003 +0000 UTC" Nov 03 20:00:26.576953 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.576834 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13493h14m38.896052772s" Nov 03 20:00:26.716358 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.716330 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:26.716546 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:26.716434 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:26.731768 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.731690 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tffb6" event={"ID":"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a","Type":"ContainerStarted","Data":"4584496f79d6ca178888660af44772fa061029d0b91271d4ce816f43c070b4ff"} Nov 03 20:00:26.734363 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.734281 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" event={"ID":"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f","Type":"ContainerStarted","Data":"af3faa9407b7346ff5c2f1886300b5fdb7853496a98153bfc21c30fa280adda6"} Nov 03 20:00:26.750874 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.750797 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8fpvs" event={"ID":"042f2f33-8a62-4464-bdd8-1a54ed18272d","Type":"ContainerStarted","Data":"d37bd4fe64f280a4e42d76022d6f13a2367dabd5d16c77277c3161b8ceaf480a"} Nov 03 20:00:26.759355 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.759304 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" event={"ID":"43579876-2c2b-4730-bd82-717f9b116bab","Type":"ContainerStarted","Data":"fedd28b690458e403ed90cc0c8fc02c863c55e5a25868849590c9dda395353c8"} Nov 03 20:00:26.764874 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.764842 2583 generic.go:356] "Generic (PLEG): container finished" podID="62881be2bc7cb00291f2b39128d27e0c" containerID="eb3d1288bbad91b647d5c022ef77fe7786337e29c6bc6e416c8b7b99873e5103" exitCode=0 Nov 03 20:00:26.765153 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.765102 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"62881be2bc7cb00291f2b39128d27e0c","Type":"ContainerDied","Data":"eb3d1288bbad91b647d5c022ef77fe7786337e29c6bc6e416c8b7b99873e5103"} Nov 03 20:00:26.782218 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.782191 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5h7vn" event={"ID":"baba1acd-b516-4149-a337-d56b9e74b36e","Type":"ContainerStarted","Data":"24033f77e98e2b98387bbff636ca30f4fbcb5c270f4e650dfba84d475b2b87e9"} Nov 03 20:00:26.795357 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.795308 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerStarted","Data":"f9c98cbda157f34909384446fce81fe2770c3486fb529684caadcc23623a97f4"} Nov 03 20:00:26.799856 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.799613 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-56d9g" event={"ID":"a3f23518-97e0-4f3f-b939-d76052dc1339","Type":"ContainerStarted","Data":"622e2d08eee35041ae23385941f7c82d5a4d4a7847d17ff0604944ba4cdf0258"} Nov 03 20:00:26.812663 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.812635 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerStarted","Data":"f0cba8e705823b06d02bad90ef7f7806afaa9ef45e6d46342e1ceaea064538fe"} Nov 03 20:00:26.818456 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:26.818416 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wszxg" event={"ID":"fbdb17a3-3ead-4c90-9922-31fbae907537","Type":"ContainerStarted","Data":"88e618455f8f22e472094ce96e7ddd4ffb1c16231f006da80f767e0977d456fe"} Nov 03 20:00:27.156919 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:27.156288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:27.156919 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.156483 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:27.156919 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.156539 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret podName:a9998949-10c8-4511-857c-dfad4001ab59 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:29.156522276 +0000 UTC m=+5.962502842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret") pod "global-pull-secret-syncer-bhl7x" (UID: "a9998949-10c8-4511-857c-dfad4001ab59") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:27.258139 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:27.257489 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:27.258139 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.257676 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:27.258139 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.257737 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs podName:dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:29.257719685 +0000 UTC m=+6.063700242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs") pod "network-metrics-daemon-xf24w" (UID: "dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:27.358828 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:27.358788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:27.359029 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.358944 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:27.359029 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.358963 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:27.359029 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.358977 2583 projected.go:196] Error preparing data for projected volume kube-api-access-wsfgx for pod openshift-network-diagnostics/network-check-target-4xx8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:27.359029 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.359030 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx podName:a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b nodeName:}" failed. No retries permitted until 2026-11-03 20:00:29.359014237 +0000 UTC m=+6.164994805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfgx" (UniqueName: "kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx") pod "network-check-target-4xx8n" (UID: "a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:27.578060 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:27.577925 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:55:25 +0000 UTC" deadline="2028-07-29 02:33:24.637846056 +0000 UTC" Nov 03 20:00:27.578060 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:27.577984 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15198h32m57.059866427s" Nov 03 20:00:27.719228 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:27.717565 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:27.719228 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.717697 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:27.719228 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:27.718179 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:27.719228 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:27.718286 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:27.825639 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:27.824896 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"62881be2bc7cb00291f2b39128d27e0c","Type":"ContainerStarted","Data":"5d55fd9976f79b0a19d7a0e989310f596dfe37db2fb20af7e58bd1f185369081"} Nov 03 20:00:28.717055 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:28.716576 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:28.717055 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:28.716704 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:29.175674 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.174983 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:29.175674 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.175203 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:29.175674 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.175268 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret podName:a9998949-10c8-4511-857c-dfad4001ab59 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:33.175250861 +0000 UTC m=+9.981231431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret") pod "global-pull-secret-syncer-bhl7x" (UID: "a9998949-10c8-4511-857c-dfad4001ab59") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:29.276477 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.276230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:29.277185 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.276674 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:29.277185 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.276767 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs podName:dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:33.276744804 +0000 UTC m=+10.082725359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs") pod "network-metrics-daemon-xf24w" (UID: "dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:29.378083 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.377577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:29.378083 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.377718 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:29.378083 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.377740 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:29.378083 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.377750 2583 projected.go:196] Error preparing data for projected volume kube-api-access-wsfgx for pod openshift-network-diagnostics/network-check-target-4xx8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:29.378083 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.377793 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx podName:a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b nodeName:}" failed. No retries permitted until 2026-11-03 20:00:33.377779127 +0000 UTC m=+10.183759679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfgx" (UniqueName: "kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx") pod "network-check-target-4xx8n" (UID: "a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:29.719261 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.719193 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:29.719695 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.719328 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:29.719763 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.719743 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:29.719868 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:29.719848 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:29.864998 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.864958 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"23259ea2648a683e23b5758ee6ae68e7","Type":"ContainerStarted","Data":"25c22b402723be6ff25dee8536aea7f0adadff42eaf9a7b8a8184384787edd07"} Nov 03 20:00:29.868821 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.868792 2583 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="5a3619123f177e8fc287dd7732e4d6a891ab35d381c16356d85c61913fcd86bd" exitCode=0 Nov 03 20:00:29.868948 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.868849 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerDied","Data":"5a3619123f177e8fc287dd7732e4d6a891ab35d381c16356d85c61913fcd86bd"} Nov 03 20:00:29.883778 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.882349 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" podStartSLOduration=4.882332027 podStartE2EDuration="4.882332027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:27.843940839 +0000 UTC m=+4.649921412" watchObservedRunningTime="2026-11-03 20:00:29.882332027 +0000 UTC m=+6.688312600" Nov 03 20:00:29.909245 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:29.908738 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" podStartSLOduration=4.90872078 podStartE2EDuration="4.90872078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:29.883597653 +0000 UTC m=+6.689578228" watchObservedRunningTime="2026-11-03 20:00:29.90872078 +0000 UTC m=+6.714701334" Nov 03 20:00:30.716991 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:30.716937 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:30.717217 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:30.717045 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:30.873967 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:30.873938 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8fpvs" event={"ID":"042f2f33-8a62-4464-bdd8-1a54ed18272d","Type":"ContainerStarted","Data":"9abb106834fb24704c6f8c19589092c2ceb69a8fa4f9e86958d0420e96823e42"} Nov 03 20:00:31.718785 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:31.718752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:31.718958 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:31.718878 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:31.719268 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:31.719251 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:31.719374 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:31.719356 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:31.879090 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:31.879051 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5h7vn" event={"ID":"baba1acd-b516-4149-a337-d56b9e74b36e","Type":"ContainerStarted","Data":"7489e524beebbc07e42c454624f57663e1fd33476a3a74bd9226501decdb1fe4"} Nov 03 20:00:31.882676 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:31.882646 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" event={"ID":"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f","Type":"ContainerStarted","Data":"501ce9ae97df8e66b8001ef0664c3842cae4b54dea23adf3c104ba487b3ca3d0"} Nov 03 20:00:31.901658 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:31.901608 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8fpvs" podStartSLOduration=4.835704305 podStartE2EDuration="8.901590001s" totalImagesPullingTime="4.065885696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:30.891782794 +0000 UTC m=+7.697763368" watchObservedRunningTime="2026-11-03 20:00:31.901590001 +0000 UTC m=+8.707570574" Nov 03 20:00:31.902008 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:31.901983 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5h7vn" podStartSLOduration=3.8656540440000002 podStartE2EDuration="8.901977922s" totalImagesPullingTime="5.036323878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:31.900661843 +0000 UTC m=+8.706642415" watchObservedRunningTime="2026-11-03 20:00:31.901977922 +0000 UTC m=+8.707958494" Nov 03 20:00:32.710223 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:32.710196 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:00:32.716238 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:32.716221 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:32.716336 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:32.716316 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:32.887745 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:32.887601 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" event={"ID":"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f","Type":"ContainerStarted","Data":"f54ea57e062a3eae525f4941d1eef7814df7c20c537467c86b1b0d1a45e0d688"} Nov 03 20:00:32.890100 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:32.890065 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-56d9g" event={"ID":"a3f23518-97e0-4f3f-b939-d76052dc1339","Type":"ContainerStarted","Data":"cb449686831d4ada2d76f5bbd5bdf3d457625402cd1a4a58bd868669757956a5"} Nov 03 20:00:33.091599 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.091575 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:33.092366 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.092348 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:33.110699 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.110660 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-56d9g" podStartSLOduration=3.287932193 podStartE2EDuration="10.110648375s" totalImagesPullingTime="6.822716182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:32.913540812 +0000 UTC m=+9.719521386" watchObservedRunningTime="2026-11-03 20:00:33.110648375 +0000 UTC m=+9.916628947" Nov 03 20:00:33.222077 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.221607 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:33.222077 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.221765 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:33.222077 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.221814 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret podName:a9998949-10c8-4511-857c-dfad4001ab59 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:41.22180107 +0000 UTC m=+18.027781622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret") pod "global-pull-secret-syncer-bhl7x" (UID: "a9998949-10c8-4511-857c-dfad4001ab59") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:33.322256 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.322225 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:33.322420 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.322331 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:33.322420 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.322388 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs podName:dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:41.322368223 +0000 UTC m=+18.128348775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs") pod "network-metrics-daemon-xf24w" (UID: "dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:33.423531 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.423438 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:33.423681 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.423577 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:33.423681 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.423599 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:33.423681 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.423611 2583 projected.go:196] Error preparing data for projected volume kube-api-access-wsfgx for pod openshift-network-diagnostics/network-check-target-4xx8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:33.423681 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.423671 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx podName:a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b nodeName:}" failed. No retries permitted until 2026-11-03 20:00:41.42365408 +0000 UTC m=+18.229634631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfgx" (UniqueName: "kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx") pod "network-check-target-4xx8n" (UID: "a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:33.599280 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.598997 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:00:32.710214026Z","UUID":"5fa79310-9748-4981-b884-b468d3a320e1","Handler":null,"Name":"","Endpoint":""} Nov 03 20:00:33.601992 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.601966 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:00:33.601992 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.601996 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:00:33.717317 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.717288 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:33.717480 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.717467 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:33.717613 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.717463 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:33.717688 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:33.717598 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:33.894518 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.894317 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerStarted","Data":"cb35a69aaee8fa08d2be748f4ba888d2623c914245259b0e7a331c67d71f00c4"} Nov 03 20:00:33.896299 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.896233 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wszxg" event={"ID":"fbdb17a3-3ead-4c90-9922-31fbae907537","Type":"ContainerStarted","Data":"91d946ff3abb77c993235f68bb79592eebb689d30f29a4950b5dc52117d4e353"} Nov 03 20:00:33.937439 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:33.937347 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wszxg" podStartSLOduration=3.828598032 podStartE2EDuration="10.93732739s" totalImagesPullingTime="7.108729358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:33.937152685 +0000 UTC m=+10.743133256" watchObservedRunningTime="2026-11-03 20:00:33.93732739 +0000 UTC m=+10.743307970" Nov 03 20:00:34.716454 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:34.716425 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:34.716593 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:34.716547 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:34.899823 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:34.899792 2583 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="cb35a69aaee8fa08d2be748f4ba888d2623c914245259b0e7a331c67d71f00c4" exitCode=0 Nov 03 20:00:34.900415 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:34.899866 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerDied","Data":"cb35a69aaee8fa08d2be748f4ba888d2623c914245259b0e7a331c67d71f00c4"} Nov 03 20:00:34.901386 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:34.901368 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tffb6" event={"ID":"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a","Type":"ContainerStarted","Data":"cb57ca584c2e20849e378d11bcfaed3585d351f9ea94e4d575a7fadd3128f94f"} Nov 03 20:00:34.903491 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:34.903470 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" event={"ID":"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f","Type":"ContainerStarted","Data":"514d21e13ef4667e314c28dce9d5f05d44ec02569a4a4b771337e9d2799a7fb3"} Nov 03 20:00:34.904820 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:34.904803 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" event={"ID":"43579876-2c2b-4730-bd82-717f9b116bab","Type":"ContainerStarted","Data":"baf68d956381cbb48ef194ec9b3cc6c5b2a667b72252026bff0c8e226dba52b0"} Nov 03 20:00:34.943566 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:34.943530 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" podStartSLOduration=3.229357887 podStartE2EDuration="11.943520415s" totalImagesPullingTime="8.714162528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:34.943186517 +0000 UTC m=+11.749167089" watchObservedRunningTime="2026-11-03 20:00:34.943520415 +0000 UTC m=+11.749500988" Nov 03 20:00:34.962803 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:34.962769 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" podStartSLOduration=4.176778374 podStartE2EDuration="11.962758946s" totalImagesPullingTime="7.785980572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:34.962629597 +0000 UTC m=+11.768610170" watchObservedRunningTime="2026-11-03 20:00:34.962758946 +0000 UTC m=+11.768739519" Nov 03 20:00:34.978962 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:34.978931 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tffb6" podStartSLOduration=4.760725498 podStartE2EDuration="11.978922541s" totalImagesPullingTime="7.218197043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:34.978781822 +0000 UTC m=+11.784762394" watchObservedRunningTime="2026-11-03 20:00:34.978922541 +0000 UTC m=+11.784903114" Nov 03 20:00:35.716611 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:35.716576 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:35.716749 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:35.716576 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:35.716749 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:35.716685 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:35.716858 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:35.716833 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:36.716796 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:36.716634 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:36.717257 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:36.716871 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:36.910416 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:36.910386 2583 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="df482d9cea138e91f6418f61d1797b9e2fe60442228303047caa102366945921" exitCode=0 Nov 03 20:00:36.910590 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:36.910459 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerDied","Data":"df482d9cea138e91f6418f61d1797b9e2fe60442228303047caa102366945921"} Nov 03 20:00:37.716165 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:37.716143 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:37.716247 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:37.716185 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:37.716298 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:37.716278 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:37.716425 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:37.716404 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:37.914668 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:37.914598 2583 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="3023f0acb6b191a70f61fa8e533f1ac3af0e80058381f6750cedfd735b2aa7cf" exitCode=0 Nov 03 20:00:37.914668 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:37.914637 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerDied","Data":"3023f0acb6b191a70f61fa8e533f1ac3af0e80058381f6750cedfd735b2aa7cf"} Nov 03 20:00:38.716130 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:38.716094 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:38.716297 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:38.716198 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:39.717068 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:39.717037 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:39.717068 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:39.717070 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:39.717501 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:39.717138 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:39.717501 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:39.717287 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:39.921864 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:39.921837 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerStarted","Data":"a1e73fe6f50ceaf6d8c7f7a5ac62a249649f99dfbc033b5629ec93b4d86e3e68"} Nov 03 20:00:40.716812 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:40.716786 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:40.717059 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:40.716890 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:40.926407 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:40.926368 2583 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="a1e73fe6f50ceaf6d8c7f7a5ac62a249649f99dfbc033b5629ec93b4d86e3e68" exitCode=0 Nov 03 20:00:40.926407 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:40.926395 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerDied","Data":"a1e73fe6f50ceaf6d8c7f7a5ac62a249649f99dfbc033b5629ec93b4d86e3e68"} Nov 03 20:00:41.284513 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:41.284473 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:41.284677 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.284583 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:41.284677 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.284658 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret podName:a9998949-10c8-4511-857c-dfad4001ab59 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.284638408 +0000 UTC m=+34.090618979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret") pod "global-pull-secret-syncer-bhl7x" (UID: "a9998949-10c8-4511-857c-dfad4001ab59") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:41.384960 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:41.384930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:41.385099 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.385065 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:41.385143 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.385117 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs podName:dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.38510321 +0000 UTC m=+34.191083766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs") pod "network-metrics-daemon-xf24w" (UID: "dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:41.485896 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:41.485868 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:41.486018 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.485970 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:41.486018 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.485982 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:41.486018 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.485991 2583 projected.go:196] Error preparing data for projected volume kube-api-access-wsfgx for pod openshift-network-diagnostics/network-check-target-4xx8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:41.486128 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.486036 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx podName:a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b nodeName:}" failed. No retries permitted until 2026-11-03 20:00:57.486021945 +0000 UTC m=+34.292002689 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfgx" (UniqueName: "kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx") pod "network-check-target-4xx8n" (UID: "a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:41.716870 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:41.716798 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:41.717019 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:41.716810 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:41.717019 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.716901 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:41.717019 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:41.716972 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:41.930622 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:41.930594 2583 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="2bea7681c057f63ffac7b2821efe7fa7eea2c3ebcec56d94714840aea8ee8322" exitCode=0 Nov 03 20:00:41.930964 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:41.930627 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerDied","Data":"2bea7681c057f63ffac7b2821efe7fa7eea2c3ebcec56d94714840aea8ee8322"} Nov 03 20:00:42.716868 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:42.716841 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:42.717044 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:42.716946 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:42.936299 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:42.936264 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" event={"ID":"d938e951-475c-47d6-8d73-f17da86400ad","Type":"ContainerStarted","Data":"536c826383295793147b57f7879bee55e337dd7f7866b8fd74b2b8afa98cf6fc"} Nov 03 20:00:42.963953 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:42.963909 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s7lv9" podStartSLOduration=9.637386027 podStartE2EDuration="19.963896168s" totalImagesPullingTime="10.326510141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:42.963401605 +0000 UTC m=+19.769382178" watchObservedRunningTime="2026-11-03 20:00:42.963896168 +0000 UTC m=+19.769876740" Nov 03 20:00:43.717538 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:43.717502 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:43.717692 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:43.717609 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:43.717692 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:43.717661 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:43.717777 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:43.717760 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:44.716261 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:44.716228 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:44.716664 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:44.716326 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:45.716677 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:45.716644 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:45.717136 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:45.716648 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:45.717136 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:45.716782 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:45.717136 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:45.716816 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:46.716589 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:46.716557 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:46.716783 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:46.716663 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:47.716952 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:47.716917 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:47.717463 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:47.717034 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:47.717463 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:47.717083 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:47.717463 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:47.717184 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:48.594505 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:48.594477 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:48.594674 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:48.594592 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:00:48.594995 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:48.594978 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:00:48.716842 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:48.716812 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:48.717003 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:48.716914 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:49.716416 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:49.716382 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:49.716416 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:49.716408 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:49.716609 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:49.716502 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:49.716648 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:49.716618 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:50.716721 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:50.716683 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:50.717099 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:50.716794 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:51.716618 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:51.716579 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:51.716805 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:51.716706 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:51.716805 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:51.716749 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:51.717225 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:51.716841 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:52.716707 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:52.716669 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:52.716890 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:52.716797 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:53.720103 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:53.719554 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:53.720103 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:53.719603 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:53.720103 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:53.719672 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:53.720103 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:53.719764 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:54.716732 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:54.716689 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:54.716923 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:54.716798 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:55.716775 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:55.716741 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:55.717216 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:55.716874 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:55.717216 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:55.716909 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:55.717216 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:55.717031 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:56.716931 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:56.716896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:56.717382 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:56.717007 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:57.004431 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:57.004388 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:57.190029 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:57.189991 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:57.307307 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:57.307227 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:57.307463 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.307398 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:57.307536 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.307478 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret podName:a9998949-10c8-4511-857c-dfad4001ab59 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:29.307459198 +0000 UTC m=+66.113439749 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret") pod "global-pull-secret-syncer-bhl7x" (UID: "a9998949-10c8-4511-857c-dfad4001ab59") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:00:57.408266 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:57.408233 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:57.408426 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.408390 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:57.408493 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.408462 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs podName:dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:29.408431938 +0000 UTC m=+66.214412503 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs") pod "network-metrics-daemon-xf24w" (UID: "dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:00:57.508763 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:57.508733 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:57.508936 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.508893 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:00:57.508936 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.508916 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:00:57.508936 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.508929 2583 projected.go:196] Error preparing data for projected volume kube-api-access-wsfgx for pod openshift-network-diagnostics/network-check-target-4xx8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:57.509075 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.508987 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx podName:a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b nodeName:}" failed. No retries permitted until 2026-11-03 20:01:29.508971271 +0000 UTC m=+66.314951838 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfgx" (UniqueName: "kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx") pod "network-check-target-4xx8n" (UID: "a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:00:57.716884 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:57.716802 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:57.717029 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:57.716806 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:57.717029 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.716971 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:57.717029 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:57.717019 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:58.716784 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:58.716745 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:00:58.716968 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:58.716884 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:00:59.105995 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.105963 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:59.237064 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.237024 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:59.716130 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.716104 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:00:59.716295 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:59.716234 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:00:59.716295 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.716288 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:00:59.716420 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:00:59.716398 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:00:59.789213 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.789188 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:59.924814 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.924787 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:00:59.969757 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.969718 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:00:59.970077 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.970055 2583 generic.go:356] "Generic (PLEG): container finished" podID="9af6745a-681c-44b6-861c-310df033cfe9" containerID="2cf9cb15ac5be549bcf98d35578c64555626a3c32c181802373ff4ce8668e538" exitCode=1 Nov 03 20:00:59.970124 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.970100 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerDied","Data":"2cf9cb15ac5be549bcf98d35578c64555626a3c32c181802373ff4ce8668e538"} Nov 03 20:00:59.970194 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:00:59.970184 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerStarted","Data":"c16a47e63e99f72ed3477753f56460f51654f0f0898816a2d488bd647b8ef905"} Nov 03 20:01:00.690206 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:00.690006 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:00.717084 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:00.717057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:00.717205 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:00.717173 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:00.974542 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:00.974476 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:01:00.974822 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:00.974793 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerStarted","Data":"08a217fda20a53368f73469ef862d8fa07ddfd7068a432e0981e0886af6aca2d"} Nov 03 20:01:00.974871 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:00.974861 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerStarted","Data":"09246361eb180e06acd330bd97a83058621dc67f2d28814f63a3d70d1b9f5fbf"} Nov 03 20:01:00.974905 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:00.974892 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerStarted","Data":"c2ba634fb21afb0658f7657a9280183b205063902995d29f6bc9de174ebce4b0"} Nov 03 20:01:00.974946 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:00.974933 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerStarted","Data":"315fea39abd7e4a871133d69075de57fd05997d0d00e95b05dc44ad5ab3e7d86"} Nov 03 20:01:01.716603 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:01.716572 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:01.717125 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:01.716663 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:01.717125 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:01.716774 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:01.717125 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:01.716898 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:01.897271 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:01.897233 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:02.036152 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:02.036124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:02.190730 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:02.190704 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:02.338949 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:02.338889 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wszxg_fbdb17a3-3ead-4c90-9922-31fbae907537/dns-node-resolver/0.log" Nov 03 20:01:02.353024 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:02.352999 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:02.485298 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:02.485270 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:02.716293 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:02.716220 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:02.716436 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:02.716319 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:02.937611 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:02.937584 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5h7vn_baba1acd-b516-4149-a337-d56b9e74b36e/node-ca/0.log" Nov 03 20:01:02.981831 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:02.981757 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:01:02.982140 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:02.982120 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerStarted","Data":"63cb7e8dbfc6196db3a47331758360ea677bb74d328f21aa2db144d6b7e2dadf"} Nov 03 20:01:03.308525 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:03.308491 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:03.646517 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:03.646430 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:03.717388 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:03.717359 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:03.717534 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:03.717507 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:03.717579 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:03.717553 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:03.717665 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:03.717648 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:03.779193 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:03.779159 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:04.717013 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:04.716851 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:04.717470 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:04.717096 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:04.989544 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:04.989518 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:01:04.989900 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:04.989869 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerStarted","Data":"a6e3b9a9824f177beafeee60e19a8e4deaa04c138e3c91e04ac4422c95d225e6"} Nov 03 20:01:04.990144 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:04.990120 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:04.990255 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:04.990193 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:04.990312 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:04.990279 2583 scope.go:122] "RemoveContainer" containerID="2cf9cb15ac5be549bcf98d35578c64555626a3c32c181802373ff4ce8668e538" Nov 03 20:01:05.005557 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:05.005517 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:05.384934 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:05.384900 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:05.509517 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:05.509488 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:05.717087 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:05.717002 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:05.717494 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:05.717134 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:05.717494 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:05.717177 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:05.717494 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:05.717263 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:05.994999 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:05.994921 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:01:05.995277 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:05.995254 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" event={"ID":"9af6745a-681c-44b6-861c-310df033cfe9","Type":"ContainerStarted","Data":"5cc14604d5e67ff65bd5a23943428a8538f6c5a1fd7cfafdcce934663718b446"} Nov 03 20:01:05.995535 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:05.995519 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:06.010493 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:06.010467 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:06.030803 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:06.030760 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" podStartSLOduration=9.136974965 podStartE2EDuration="43.030747085s" totalImagesPullingTime="33.89377212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:06.029161773 +0000 UTC m=+42.835142345" watchObservedRunningTime="2026-11-03 20:01:06.030747085 +0000 UTC m=+42.836727655" Nov 03 20:01:06.510577 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:06.510543 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:06.717014 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:06.716976 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:06.717151 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:06.717080 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:07.716896 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:07.716870 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:07.717046 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:07.716969 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:07.717046 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:07.716998 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:07.717138 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:07.717043 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:08.170947 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:08.170913 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:08.292011 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:08.291967 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:08.681155 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:08.681124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:08.716573 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:08.716551 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:08.716714 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:08.716648 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:08.854290 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:08.854257 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:08.984146 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:08.984072 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:09.127319 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:09.127284 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:09.716908 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:09.716877 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:09.717297 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:09.716985 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:09.717297 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:09.717074 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:09.717297 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:09.717194 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:10.325634 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:10.325599 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:10.518999 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:10.518963 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:10.716157 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:10.716076 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:10.716302 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:10.716176 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:10.720119 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:10.720096 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:10.918731 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:10.918693 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:11.122047 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:11.122010 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:11.560643 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:11.560612 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:11.716416 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:11.716381 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:11.716590 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:11.716538 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:11.716590 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:11.716583 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:11.716681 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:11.716654 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:11.959469 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:11.959386 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:12.317824 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:12.317787 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:12.557794 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:12.557739 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:12.716390 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:12.716309 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:12.716554 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:12.716414 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:12.722502 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:12.722476 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:12.925792 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:12.925765 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:13.054126 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:13.054076 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:13.188428 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:13.188395 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:13.716985 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:13.716957 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:13.717145 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:13.717057 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:13.717145 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:13.717119 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:13.717625 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:13.717596 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:14.120377 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:14.120346 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:14.523493 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:14.523433 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:14.716193 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:14.716162 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:14.716356 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:14.716260 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:14.720309 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:14.720289 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:14.921794 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:14.921711 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:15.269431 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:15.269405 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:15.406824 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:15.406794 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:15.716921 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:15.716841 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:15.717068 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:15.716956 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:15.717068 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:15.717011 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:15.717150 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:15.717104 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:16.127481 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:16.127422 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:16.524425 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:16.524386 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:16.654672 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:16.654643 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:16.716639 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:16.716599 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:16.716780 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:16.716718 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:16.785081 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:16.784983 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:16.918685 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:16.918655 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:17.320438 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:17.320410 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:17.718867 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:17.718800 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:17.719250 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:17.718916 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:17.719250 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:17.718965 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:17.719250 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:17.719047 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:18.023065 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:18.023040 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:01:18.023187 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:18.023080 2583 generic.go:356] "Generic (PLEG): container finished" podID="a3f23518-97e0-4f3f-b939-d76052dc1339" containerID="cb449686831d4ada2d76f5bbd5bdf3d457625402cd1a4a58bd868669757956a5" exitCode=1 Nov 03 20:01:18.023187 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:18.023127 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-56d9g" event={"ID":"a3f23518-97e0-4f3f-b939-d76052dc1339","Type":"ContainerDied","Data":"cb449686831d4ada2d76f5bbd5bdf3d457625402cd1a4a58bd868669757956a5"} Nov 03 20:01:18.023508 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:18.023492 2583 scope.go:122] "RemoveContainer" containerID="cb449686831d4ada2d76f5bbd5bdf3d457625402cd1a4a58bd868669757956a5" Nov 03 20:01:18.122524 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:18.122342 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:18.524685 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:18.524651 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:18.716613 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:18.716574 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:18.716775 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:18.716680 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:18.725219 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:18.725199 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:18.919335 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:18.919256 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:19.026779 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:19.026754 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:01:19.026932 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:19.026818 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-56d9g" event={"ID":"a3f23518-97e0-4f3f-b939-d76052dc1339","Type":"ContainerStarted","Data":"c35bed4a09701107a848532f0d93ddf969790f17da643ef38178c4e425c3bdd5"} Nov 03 20:01:19.122590 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:19.122560 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:19.716197 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:19.716166 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:19.716364 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:19.716166 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:19.716364 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:19.716282 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:19.716364 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:19.716318 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:20.716378 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:20.716351 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:20.716760 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:20.716457 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:21.155298 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:21.155265 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:21.354947 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:21.354914 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:21.716781 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:21.716742 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:21.717276 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:21.716880 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:21.717276 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:21.717135 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:21.717276 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:21.717231 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:22.125192 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:22.125157 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bhl7x"] Nov 03 20:01:22.125338 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:22.125281 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:22.125422 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:22.125395 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:22.128535 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:22.128500 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xx8n"] Nov 03 20:01:22.128756 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:22.128726 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:22.128875 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:22.128850 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:22.130277 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:22.130239 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xf24w"] Nov 03 20:01:22.130368 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:22.130347 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:22.130495 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:22.130474 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:23.168856 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:23.168821 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:23.291877 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:23.291846 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:23.719653 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:23.719572 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:23.719653 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:23.719592 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:23.719851 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:23.719701 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:23.719851 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:23.719754 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:23.719956 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:23.719855 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:23.719956 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:23.719887 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:23.854953 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:23.854921 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:25.716644 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:25.716475 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:25.717141 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:25.716481 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:25.717141 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:25.716719 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4xx8n" podUID="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" Nov 03 20:01:25.717141 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:25.716822 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bhl7x" podUID="a9998949-10c8-4511-857c-dfad4001ab59" Nov 03 20:01:25.717141 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:25.716522 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:25.717141 ip-10-0-142-5 kubenswrapper[2583]: E1103 20:01:25.716920 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xf24w" podUID="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" Nov 03 20:01:26.495927 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.495835 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeReady" Nov 03 20:01:26.496062 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.495956 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 20:01:26.548039 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.548007 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c8hzq"] Nov 03 20:01:26.551838 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.551816 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-znlzm"] Nov 03 20:01:26.551958 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.551936 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.555547 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.555524 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.555953 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.555930 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vkxg\"" Nov 03 20:01:26.556067 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.555930 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:01:26.556067 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.556024 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:01:26.561656 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.561629 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c8hzq"] Nov 03 20:01:26.564851 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.564756 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-znlzm"] Nov 03 20:01:26.610709 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.610672 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e75b735-0229-429d-927f-5683bcd373d6-metrics-tls\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.610891 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.610726 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e75b735-0229-429d-927f-5683bcd373d6-tmp-dir\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.610891 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.610788 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92b5617d-95cb-44c2-84b5-a73a50c2d749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.610891 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.610833 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr474\" (UniqueName: \"kubernetes.io/projected/7e75b735-0229-429d-927f-5683bcd373d6-kube-api-access-tr474\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.610891 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.610850 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92b5617d-95cb-44c2-84b5-a73a50c2d749-data-volume\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.610891 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.610868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptlpz\" (UniqueName: \"kubernetes.io/projected/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-api-access-ptlpz\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.611046 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.610990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92b5617d-95cb-44c2-84b5-a73a50c2d749-crio-socket\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.611046 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.611022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.611046 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.611042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e75b735-0229-429d-927f-5683bcd373d6-config-volume\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.649106 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.649073 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7wjnx"] Nov 03 20:01:26.652319 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.652303 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:26.656048 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.656020 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:01:26.656168 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.656086 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:01:26.656168 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.656105 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mszh\"" Nov 03 20:01:26.656168 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.656097 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:01:26.656168 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.656123 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:01:26.667360 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.663901 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7wjnx"] Nov 03 20:01:26.711823 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.711771 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92b5617d-95cb-44c2-84b5-a73a50c2d749-crio-socket\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.711823 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.711824 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91a2c232-0afd-43ed-a457-c977e4961416-cert\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:26.711823 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.711843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.712114 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712035 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e75b735-0229-429d-927f-5683bcd373d6-config-volume\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.712114 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712028 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92b5617d-95cb-44c2-84b5-a73a50c2d749-crio-socket\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.712114 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712087 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e75b735-0229-429d-927f-5683bcd373d6-metrics-tls\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.712252 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e75b735-0229-429d-927f-5683bcd373d6-tmp-dir\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.712252 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712167 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92b5617d-95cb-44c2-84b5-a73a50c2d749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.712252 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712197 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djpvm\" (UniqueName: \"kubernetes.io/projected/91a2c232-0afd-43ed-a457-c977e4961416-kube-api-access-djpvm\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:26.712393 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712251 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr474\" (UniqueName: \"kubernetes.io/projected/7e75b735-0229-429d-927f-5683bcd373d6-kube-api-access-tr474\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.712393 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712282 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92b5617d-95cb-44c2-84b5-a73a50c2d749-data-volume\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.712393 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712341 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptlpz\" (UniqueName: \"kubernetes.io/projected/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-api-access-ptlpz\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.712558 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712541 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92b5617d-95cb-44c2-84b5-a73a50c2d749-data-volume\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.712680 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712662 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.712749 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712663 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e75b735-0229-429d-927f-5683bcd373d6-config-volume\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.712749 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.712713 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e75b735-0229-429d-927f-5683bcd373d6-tmp-dir\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.716230 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.716201 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92b5617d-95cb-44c2-84b5-a73a50c2d749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.716373 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.716245 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e75b735-0229-429d-927f-5683bcd373d6-metrics-tls\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.724410 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.724380 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr474\" (UniqueName: \"kubernetes.io/projected/7e75b735-0229-429d-927f-5683bcd373d6-kube-api-access-tr474\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.724807 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.724639 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptlpz\" (UniqueName: \"kubernetes.io/projected/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-api-access-ptlpz\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.758879 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.758849 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:01:26.758879 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.758849 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:01:26.758879 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.758849 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lrvk5\"" Nov 03 20:01:26.759089 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.758925 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:01:26.813693 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.813662 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91a2c232-0afd-43ed-a457-c977e4961416-cert\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:26.813854 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.813714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djpvm\" (UniqueName: \"kubernetes.io/projected/91a2c232-0afd-43ed-a457-c977e4961416-kube-api-access-djpvm\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:26.816069 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.816043 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91a2c232-0afd-43ed-a457-c977e4961416-cert\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:26.826013 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.825988 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djpvm\" (UniqueName: \"kubernetes.io/projected/91a2c232-0afd-43ed-a457-c977e4961416-kube-api-access-djpvm\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:26.870917 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.870882 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:26.879310 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.879281 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:26.963843 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:26.963812 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:27.157748 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.157715 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c8hzq"] Nov 03 20:01:27.161191 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:01:27.161164 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e75b735_0229_429d_927f_5683bcd373d6.slice/crio-0f300d4c9fab06761d5bc3c414c322c1070d0277d3b98e626d704a9233539a89 WatchSource:0}: Error finding container 0f300d4c9fab06761d5bc3c414c322c1070d0277d3b98e626d704a9233539a89: Status 404 returned error can't find the container with id 0f300d4c9fab06761d5bc3c414c322c1070d0277d3b98e626d704a9233539a89 Nov 03 20:01:27.161492 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.161431 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-znlzm"] Nov 03 20:01:27.164325 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:01:27.164299 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b5617d_95cb_44c2_84b5_a73a50c2d749.slice/crio-847954b04a80309354f4c24463571bb8ede786c010c70c68fe5680c9270de4ac WatchSource:0}: Error finding container 847954b04a80309354f4c24463571bb8ede786c010c70c68fe5680c9270de4ac: Status 404 returned error can't find the container with id 847954b04a80309354f4c24463571bb8ede786c010c70c68fe5680c9270de4ac Nov 03 20:01:27.233590 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.233563 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7wjnx"] Nov 03 20:01:27.236990 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:01:27.236958 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91a2c232_0afd_43ed_a457_c977e4961416.slice/crio-886cf21afb9cf752cdb9b52f66d860a9b8896e7869d1ba8fdcfe2209f99470ac WatchSource:0}: Error finding container 886cf21afb9cf752cdb9b52f66d860a9b8896e7869d1ba8fdcfe2209f99470ac: Status 404 returned error can't find the container with id 886cf21afb9cf752cdb9b52f66d860a9b8896e7869d1ba8fdcfe2209f99470ac Nov 03 20:01:27.721390 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.721360 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:27.721603 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.721581 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:27.721959 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.721727 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:27.725534 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.725511 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:01:27.725936 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.725714 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:01:27.727115 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.727089 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zc6h4\"" Nov 03 20:01:27.727115 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.727100 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:01:27.727215 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.727190 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gz7vs\"" Nov 03 20:01:27.727215 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.727190 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nc2g5\"" Nov 03 20:01:27.728232 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:27.728214 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:01:28.049186 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.048962 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7wjnx" event={"ID":"91a2c232-0afd-43ed-a457-c977e4961416","Type":"ContainerStarted","Data":"886cf21afb9cf752cdb9b52f66d860a9b8896e7869d1ba8fdcfe2209f99470ac"} Nov 03 20:01:28.050950 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.050918 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znlzm" event={"ID":"92b5617d-95cb-44c2-84b5-a73a50c2d749","Type":"ContainerStarted","Data":"c5427c5065593b1cb4b72ce2bef80e465e5beca088de00922def6eb1945b23e3"} Nov 03 20:01:28.051087 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.050980 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znlzm" event={"ID":"92b5617d-95cb-44c2-84b5-a73a50c2d749","Type":"ContainerStarted","Data":"847954b04a80309354f4c24463571bb8ede786c010c70c68fe5680c9270de4ac"} Nov 03 20:01:28.052352 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.052325 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c8hzq" event={"ID":"7e75b735-0229-429d-927f-5683bcd373d6","Type":"ContainerStarted","Data":"0f300d4c9fab06761d5bc3c414c322c1070d0277d3b98e626d704a9233539a89"} Nov 03 20:01:28.367249 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.367212 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:28.527630 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.526659 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb"] Nov 03 20:01:28.529902 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.529871 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:28.534756 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.534730 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 20:01:28.534883 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.534779 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7gvxc\"" Nov 03 20:01:28.543115 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.543050 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb"] Nov 03 20:01:28.550619 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.550594 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:28.627509 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.627396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd19fe74-5279-4b82-90e2-37bee00f7ccc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hh4rb\" (UID: \"cd19fe74-5279-4b82-90e2-37bee00f7ccc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:28.671866 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.671820 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:28.728741 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.728702 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd19fe74-5279-4b82-90e2-37bee00f7ccc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hh4rb\" (UID: \"cd19fe74-5279-4b82-90e2-37bee00f7ccc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:28.733289 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.733258 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd19fe74-5279-4b82-90e2-37bee00f7ccc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hh4rb\" (UID: \"cd19fe74-5279-4b82-90e2-37bee00f7ccc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:28.798090 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.798039 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:28.843133 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:28.843095 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:29.061804 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.061764 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znlzm" event={"ID":"92b5617d-95cb-44c2-84b5-a73a50c2d749","Type":"ContainerStarted","Data":"c08a3d923ea9596f820d2afcb1bb3a84c6b52630e165b74335faa9dfaa0cd27b"} Nov 03 20:01:29.127941 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.127899 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb"] Nov 03 20:01:29.132349 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:01:29.132311 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd19fe74_5279_4b82_90e2_37bee00f7ccc.slice/crio-2c50c65d22129c93e1ee1764729642e2057307959ce8bc58c62c0c2bf0044439 WatchSource:0}: Error finding container 2c50c65d22129c93e1ee1764729642e2057307959ce8bc58c62c0c2bf0044439: Status 404 returned error can't find the container with id 2c50c65d22129c93e1ee1764729642e2057307959ce8bc58c62c0c2bf0044439 Nov 03 20:01:29.333209 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.332988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:29.336491 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.336460 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:29.434807 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.434272 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:29.437196 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.437168 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:29.534775 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.534736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:29.537879 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.537847 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:29.547505 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.547474 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:29.555485 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.555230 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:29.563809 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.563695 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:29.848884 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.846000 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bhl7x"] Nov 03 20:01:29.848884 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.847272 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4xx8n"] Nov 03 20:01:29.848884 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:01:29.848593 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9998949_10c8_4511_857c_dfad4001ab59.slice/crio-0a700b06825ad6ae433ac671c5b872229d064607749f41d72c7a26353abcd058 WatchSource:0}: Error finding container 0a700b06825ad6ae433ac671c5b872229d064607749f41d72c7a26353abcd058: Status 404 returned error can't find the container with id 0a700b06825ad6ae433ac671c5b872229d064607749f41d72c7a26353abcd058 Nov 03 20:01:29.850614 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:01:29.850592 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a36f2d_4a81_45b5_92b3_cbf2d69a2b6b.slice/crio-dec9ccdd820d474ec01255a4bece78c0cf893625b5b1f56a7b69ac0e856ff216 WatchSource:0}: Error finding container dec9ccdd820d474ec01255a4bece78c0cf893625b5b1f56a7b69ac0e856ff216: Status 404 returned error can't find the container with id dec9ccdd820d474ec01255a4bece78c0cf893625b5b1f56a7b69ac0e856ff216 Nov 03 20:01:29.878677 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:29.878646 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xf24w"] Nov 03 20:01:29.882879 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:01:29.882839 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9b08ea_b7e2_4d9f_8e6a_2c363d1a8969.slice/crio-d486cf1ec6c7166adcdee9f01f30056eaf1e40ea2510484e4da6b9a37905ec6c WatchSource:0}: Error finding container d486cf1ec6c7166adcdee9f01f30056eaf1e40ea2510484e4da6b9a37905ec6c: Status 404 returned error can't find the container with id d486cf1ec6c7166adcdee9f01f30056eaf1e40ea2510484e4da6b9a37905ec6c Nov 03 20:01:30.066483 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.066387 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c8hzq" event={"ID":"7e75b735-0229-429d-927f-5683bcd373d6","Type":"ContainerStarted","Data":"47ddb9476e64ddd5262a8febeb30e8945d0e425433d8749a4039dacd9b796b28"} Nov 03 20:01:30.066483 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.066468 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c8hzq" event={"ID":"7e75b735-0229-429d-927f-5683bcd373d6","Type":"ContainerStarted","Data":"bd071042e506060d6b2d4c437dfde3550a7dde127f277674e73804ddbfc3f9a1"} Nov 03 20:01:30.066691 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.066503 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:30.068252 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.068224 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xf24w" event={"ID":"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969","Type":"ContainerStarted","Data":"d486cf1ec6c7166adcdee9f01f30056eaf1e40ea2510484e4da6b9a37905ec6c"} Nov 03 20:01:30.069584 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.069557 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xx8n" event={"ID":"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b","Type":"ContainerStarted","Data":"dec9ccdd820d474ec01255a4bece78c0cf893625b5b1f56a7b69ac0e856ff216"} Nov 03 20:01:30.071370 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.071320 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bhl7x" event={"ID":"a9998949-10c8-4511-857c-dfad4001ab59","Type":"ContainerStarted","Data":"0a700b06825ad6ae433ac671c5b872229d064607749f41d72c7a26353abcd058"} Nov 03 20:01:30.073686 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.073649 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-znlzm" event={"ID":"92b5617d-95cb-44c2-84b5-a73a50c2d749","Type":"ContainerStarted","Data":"f087cdf514db314050178bc0c9cb92e1c364145b1d5e65fe7c802908b90e9a9e"} Nov 03 20:01:30.075033 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.074966 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" event={"ID":"cd19fe74-5279-4b82-90e2-37bee00f7ccc","Type":"ContainerStarted","Data":"2c50c65d22129c93e1ee1764729642e2057307959ce8bc58c62c0c2bf0044439"} Nov 03 20:01:30.089125 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.089081 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-c8hzq" podStartSLOduration=2.233707461 podStartE2EDuration="4.089069078s" totalImagesPullingTime="1.855361617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:30.088065272 +0000 UTC m=+66.894045845" watchObservedRunningTime="2026-11-03 20:01:30.089069078 +0000 UTC m=+66.895049650" Nov 03 20:01:30.109564 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:30.109510 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-znlzm" podStartSLOduration=1.638287637 podStartE2EDuration="4.10949733s" totalImagesPullingTime="2.471209693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:01:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:30.10874121 +0000 UTC m=+66.914721784" watchObservedRunningTime="2026-11-03 20:01:30.10949733 +0000 UTC m=+66.915477905" Nov 03 20:01:31.083889 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.083676 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7wjnx" event={"ID":"91a2c232-0afd-43ed-a457-c977e4961416","Type":"ContainerStarted","Data":"e6b0b5d6923041fe55ee874d3f227e45546558b226f9c145e5345ceef36b1117"} Nov 03 20:01:31.087823 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.087748 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" event={"ID":"cd19fe74-5279-4b82-90e2-37bee00f7ccc","Type":"ContainerStarted","Data":"0297e7aa4252879e727b805c3db42743eb85b34b6c7eab47a6eeb92c682222ac"} Nov 03 20:01:31.088694 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.088619 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:31.094886 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.094836 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:31.129016 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.128956 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" podStartSLOduration=1.301159265 podStartE2EDuration="3.128939575s" totalImagesPullingTime="1.82778031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:31.108501941 +0000 UTC m=+67.914482517" watchObservedRunningTime="2026-11-03 20:01:31.128939575 +0000 UTC m=+67.934920149" Nov 03 20:01:31.608186 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.606499 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5"] Nov 03 20:01:31.608186 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.607023 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:31.641575 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.641543 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5"] Nov 03 20:01:31.641737 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.641708 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.651153 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.651018 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:01:31.651153 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.651110 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:01:31.651276 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.651175 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:01:31.678191 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.678165 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 20:01:31.678404 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.678378 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kcrwf\"" Nov 03 20:01:31.678404 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.678386 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 20:01:31.747767 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.747738 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffacfcf-65b1-4809-8930-3c2952a90fc8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.747960 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.747845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bm4n\" (UniqueName: \"kubernetes.io/projected/2ffacfcf-65b1-4809-8930-3c2952a90fc8-kube-api-access-4bm4n\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.747960 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.747935 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.748091 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.747961 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.748149 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.748087 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:31.849296 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.849264 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bm4n\" (UniqueName: \"kubernetes.io/projected/2ffacfcf-65b1-4809-8930-3c2952a90fc8-kube-api-access-4bm4n\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.849500 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.849311 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.849500 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.849340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.849726 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.849699 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffacfcf-65b1-4809-8930-3c2952a90fc8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.850414 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.850390 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffacfcf-65b1-4809-8930-3c2952a90fc8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.853762 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.853710 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.853891 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.853760 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.876857 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.876801 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bm4n\" (UniqueName: \"kubernetes.io/projected/2ffacfcf-65b1-4809-8930-3c2952a90fc8-kube-api-access-4bm4n\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:31.986159 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:31.986129 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:32.023367 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:32.023330 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:32.114688 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:32.114637 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7wjnx" podStartSLOduration=2.377018083 podStartE2EDuration="6.114619794s" totalImagesPullingTime="3.737601711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:32.113849057 +0000 UTC m=+68.919829631" watchObservedRunningTime="2026-11-03 20:01:32.114619794 +0000 UTC m=+68.920600364" Nov 03 20:01:32.172379 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:32.170476 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:32.377204 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:32.377162 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5"] Nov 03 20:01:32.380004 ip-10-0-142-5 kubenswrapper[2583]: W1103 20:01:32.379976 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffacfcf_65b1_4809_8930_3c2952a90fc8.slice/crio-7602a9a2d9313ba1c9e42dbc0b83494574da98dad0388c9b86ec067c38125392 WatchSource:0}: Error finding container 7602a9a2d9313ba1c9e42dbc0b83494574da98dad0388c9b86ec067c38125392: Status 404 returned error can't find the container with id 7602a9a2d9313ba1c9e42dbc0b83494574da98dad0388c9b86ec067c38125392 Nov 03 20:01:32.584215 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:32.584178 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:32.709956 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:32.709850 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:32.836399 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:32.836336 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:32.974674 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:32.974601 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:33.096051 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.095920 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" event={"ID":"2ffacfcf-65b1-4809-8930-3c2952a90fc8","Type":"ContainerStarted","Data":"7602a9a2d9313ba1c9e42dbc0b83494574da98dad0388c9b86ec067c38125392"} Nov 03 20:01:33.099460 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.099408 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xf24w" event={"ID":"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969","Type":"ContainerStarted","Data":"728b29e785693e0733c099c090f8b702ee604e201717024ccdc6c4383970926f"} Nov 03 20:01:33.099620 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.099503 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xf24w" event={"ID":"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969","Type":"ContainerStarted","Data":"2baed1881d3fd5ab395cb2ff028e833e39ccc3a13d3e1b812c26358148f3154e"} Nov 03 20:01:33.108520 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.108494 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:33.120454 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.120409 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xf24w" podStartSLOduration=67.909879654 podStartE2EDuration="1m10.120394297s" totalImagesPullingTime="2.210514643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:33.119688387 +0000 UTC m=+69.925668962" watchObservedRunningTime="2026-11-03 20:01:33.120394297 +0000 UTC m=+69.926374869" Nov 03 20:01:33.240628 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.240553 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:33.841816 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.841778 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-smhql"] Nov 03 20:01:33.859241 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.859210 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-smhql"] Nov 03 20:01:33.859389 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.859338 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:33.865583 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.865560 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jrhnz\"" Nov 03 20:01:33.865723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.865635 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:01:33.865723 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.865676 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:01:33.965667 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.965634 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e17c916-ecd2-4ccd-bfeb-451380f47d30-tmp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:33.965820 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:33.965678 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svzp\" (UniqueName: \"kubernetes.io/projected/1e17c916-ecd2-4ccd-bfeb-451380f47d30-kube-api-access-4svzp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:34.066415 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:34.066372 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svzp\" (UniqueName: \"kubernetes.io/projected/1e17c916-ecd2-4ccd-bfeb-451380f47d30-kube-api-access-4svzp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:34.066585 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:34.066470 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e17c916-ecd2-4ccd-bfeb-451380f47d30-tmp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:34.066802 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:34.066786 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e17c916-ecd2-4ccd-bfeb-451380f47d30-tmp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:34.080841 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:34.080820 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svzp\" (UniqueName: \"kubernetes.io/projected/1e17c916-ecd2-4ccd-bfeb-451380f47d30-kube-api-access-4svzp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:34.194172 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:34.194110 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:34.701420 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:34.701162 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-smhql"] Nov 03 20:01:34.830714 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:34.830674 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:34.971202 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:34.971124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:35.111355 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:35.111315 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" event={"ID":"2ffacfcf-65b1-4809-8930-3c2952a90fc8","Type":"ContainerStarted","Data":"4acde3250a6e6be47d498848777ce4c77c2d8d0bc1b2f30853d8a2e81ce83c6d"} Nov 03 20:01:35.111543 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:35.111378 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" event={"ID":"2ffacfcf-65b1-4809-8930-3c2952a90fc8","Type":"ContainerStarted","Data":"b482d63b4b1b455f6034b115243a710b9b23a342d5511a3a4fe729e2dd3d11d0"} Nov 03 20:01:35.112958 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:35.112931 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-smhql" event={"ID":"1e17c916-ecd2-4ccd-bfeb-451380f47d30","Type":"ContainerStarted","Data":"aa3f18d1385fd1555d70858586142eee2f8e55186c0161a49f9b096aecb56b3c"} Nov 03 20:01:35.114437 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:35.114414 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4xx8n" event={"ID":"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b","Type":"ContainerStarted","Data":"173d236fb22585fc82f95c86ae445bd218f2b79ba559bb57b2d49196605b2859"} Nov 03 20:01:35.114587 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:35.114570 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:35.131348 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:35.131298 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" podStartSLOduration=2.09622752 podStartE2EDuration="4.131286349s" totalImagesPullingTime="2.035058829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:35.130263384 +0000 UTC m=+71.936243969" watchObservedRunningTime="2026-11-03 20:01:35.131286349 +0000 UTC m=+71.937266920" Nov 03 20:01:35.150281 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:35.150232 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4xx8n" podStartSLOduration=67.575635927 podStartE2EDuration="1m12.15021481s" totalImagesPullingTime="4.574578883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:35.149650727 +0000 UTC m=+71.955631301" watchObservedRunningTime="2026-11-03 20:01:35.15021481 +0000 UTC m=+71.956195384" Nov 03 20:01:35.724940 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:35.724911 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c8hzq_7e75b735-0229-429d-927f-5683bcd373d6/dns/0.log" Nov 03 20:01:35.924508 ip-10-0-142-5 kubenswrapper[2583]: I1103 20:01:35.924478 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c8hzq_7e75b735-0229-429d-927f-5683bcd373d6/kube-rbac-proxy/0.log" Nov 03 20:01:36.495233 ip-10-0-142-5 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 20:01:36.510126 ip-10-0-142-5 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 20:01:36.510342 ip-10-0-142-5 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 20:01:36.510566 ip-10-0-142-5 systemd[1]: kubelet.service: Consumed 3.639s CPU time. Nov 03 20:01:36.520053 ip-10-0-142-5 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:01:36.579087 ip-10-0-142-5 kubenswrapper[4711]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:01:36.579087 ip-10-0-142-5 kubenswrapper[4711]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:01:36.579087 ip-10-0-142-5 kubenswrapper[4711]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:01:36.579087 ip-10-0-142-5 kubenswrapper[4711]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:01:36.582037 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.581960 4711 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:36.582037 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582033 4711 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:36.582037 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582037 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:36.582037 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582039 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:36.582037 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582042 4711 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:36.582037 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582045 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582048 4711 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582051 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582054 4711 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582056 4711 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582059 4711 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582062 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582064 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582067 4711 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582069 4711 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582072 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582074 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582077 4711 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582079 4711 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582082 4711 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582085 4711 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582087 4711 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582090 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582092 4711 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:36.582263 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582095 4711 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582097 4711 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582100 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582102 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582105 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582107 4711 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582110 4711 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582112 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582115 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582118 4711 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582120 4711 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582123 4711 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582125 4711 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582128 4711 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582130 4711 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582134 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582137 4711 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582139 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582142 4711 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582144 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:36.582746 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582147 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582150 4711 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582152 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582155 4711 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582157 4711 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582160 4711 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582162 4711 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582164 4711 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582167 4711 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582170 4711 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582172 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582174 4711 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582177 4711 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582179 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582182 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582184 4711 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582188 4711 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582192 4711 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582196 4711 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582200 4711 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:36.583275 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582205 4711 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582209 4711 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582212 4711 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582215 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582217 4711 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582220 4711 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582222 4711 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582225 4711 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582227 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582229 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582232 4711 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582234 4711 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582237 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582239 4711 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582242 4711 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582244 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582247 4711 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582249 4711 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582252 4711 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582256 4711 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:36.583769 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582258 4711 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582261 4711 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582263 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582266 4711 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582268 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582271 4711 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582274 4711 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582276 4711 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582279 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582281 4711 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582283 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582286 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582289 4711 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582292 4711 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582295 4711 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582297 4711 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582300 4711 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582303 4711 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582305 4711 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:36.584261 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582308 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582310 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.582314 4711 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583018 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583024 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583027 4711 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583030 4711 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583033 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583035 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583038 4711 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583040 4711 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583043 4711 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583045 4711 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583048 4711 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583051 4711 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583054 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583056 4711 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583059 4711 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583061 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583064 4711 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:36.584740 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583066 4711 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583069 4711 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583072 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583075 4711 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583078 4711 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583080 4711 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583083 4711 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583086 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583088 4711 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583091 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583093 4711 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583096 4711 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583098 4711 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583101 4711 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583103 4711 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583106 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583108 4711 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583111 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583113 4711 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:36.585221 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583116 4711 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583118 4711 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583121 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583123 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583126 4711 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583128 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583131 4711 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583134 4711 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583137 4711 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583139 4711 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583142 4711 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583144 4711 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583147 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583150 4711 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583155 4711 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583158 4711 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583162 4711 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583165 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583168 4711 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583171 4711 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:36.585708 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583174 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583177 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583180 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583183 4711 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583186 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583189 4711 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583191 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583194 4711 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583196 4711 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583199 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583201 4711 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583204 4711 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583206 4711 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583209 4711 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583211 4711 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583214 4711 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583216 4711 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583219 4711 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583221 4711 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583224 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:36.586193 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583227 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583230 4711 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583235 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583237 4711 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583240 4711 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583242 4711 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583245 4711 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583247 4711 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583250 4711 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583253 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583255 4711 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583258 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583260 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583263 4711 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583267 4711 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583270 4711 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583273 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583275 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583278 4711 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:36.586689 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583281 4711 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583283 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583286 4711 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583288 4711 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583291 4711 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583293 4711 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583296 4711 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583298 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583302 4711 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583306 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583308 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583792 4711 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583797 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583800 4711 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583803 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583806 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583809 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583811 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583814 4711 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:36.587153 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583816 4711 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583819 4711 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583821 4711 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583824 4711 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583826 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583828 4711 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583831 4711 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583833 4711 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583836 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583839 4711 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583842 4711 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583844 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583846 4711 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583849 4711 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583851 4711 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583854 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583856 4711 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583859 4711 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583861 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583864 4711 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:36.587631 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583866 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583869 4711 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583872 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583874 4711 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583877 4711 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583879 4711 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583882 4711 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583885 4711 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583888 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583890 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583893 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583896 4711 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583898 4711 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583901 4711 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583903 4711 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583905 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583909 4711 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583912 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583915 4711 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:36.588114 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583918 4711 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583920 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583923 4711 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583925 4711 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583928 4711 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583930 4711 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583933 4711 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583935 4711 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583937 4711 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583939 4711 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583942 4711 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583944 4711 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583947 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583949 4711 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583951 4711 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583954 4711 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583957 4711 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583959 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583962 4711 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:36.588615 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583964 4711 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583967 4711 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583970 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583973 4711 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583975 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583978 4711 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583980 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583983 4711 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583986 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583989 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583991 4711 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583993 4711 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583996 4711 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.583998 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584001 4711 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584003 4711 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584005 4711 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584008 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584011 4711 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584014 4711 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:36.589073 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584017 4711 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584020 4711 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584022 4711 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584025 4711 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584027 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584030 4711 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584032 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584035 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584039 4711 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584042 4711 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584044 4711 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584047 4711 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584050 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584053 4711 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584055 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584058 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584060 4711 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584063 4711 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584065 4711 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584067 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:36.589643 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584155 4711 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584162 4711 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584167 4711 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584171 4711 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584175 4711 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584178 4711 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584183 4711 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584187 4711 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584190 4711 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584192 4711 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584196 4711 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584199 4711 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584202 4711 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584205 4711 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584207 4711 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584210 4711 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584213 4711 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584216 4711 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584220 4711 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584223 4711 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584232 4711 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584235 4711 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584239 4711 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584243 4711 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:01:36.590127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584245 4711 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584248 4711 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584252 4711 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584255 4711 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584258 4711 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584261 4711 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584265 4711 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584268 4711 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584272 4711 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584275 4711 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584278 4711 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584281 4711 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584284 4711 flags.go:64] FLAG: --enable-server="true" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584287 4711 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584290 4711 flags.go:64] FLAG: --event-burst="100" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584293 4711 flags.go:64] FLAG: --event-qps="50" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584296 4711 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584299 4711 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584302 4711 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584306 4711 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584309 4711 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584312 4711 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584315 4711 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584318 4711 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584321 4711 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:01:36.590719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584323 4711 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584326 4711 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584329 4711 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584335 4711 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584338 4711 flags.go:64] FLAG: --feature-gates="" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584341 4711 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584345 4711 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584348 4711 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584352 4711 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584355 4711 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584358 4711 flags.go:64] FLAG: --help="false" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584361 4711 flags.go:64] FLAG: --hostname-override="ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584365 4711 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584368 4711 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584370 4711 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584374 4711 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584377 4711 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584380 4711 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584382 4711 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584385 4711 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584388 4711 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584391 4711 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584394 4711 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584397 4711 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:01:36.591317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584400 4711 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584403 4711 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584406 4711 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584408 4711 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584411 4711 flags.go:64] FLAG: --lock-file="" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584414 4711 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584417 4711 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584420 4711 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584426 4711 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584428 4711 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584431 4711 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584437 4711 flags.go:64] FLAG: --logging-format="text" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584439 4711 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584458 4711 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584461 4711 flags.go:64] FLAG: --manifest-url="" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584464 4711 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584468 4711 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584472 4711 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584475 4711 flags.go:64] FLAG: --max-pods="110" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584478 4711 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584481 4711 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584486 4711 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584489 4711 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584492 4711 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584495 4711 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:01:36.591913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584498 4711 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584504 4711 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584507 4711 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584510 4711 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584513 4711 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584516 4711 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584519 4711 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584522 4711 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584525 4711 flags.go:64] FLAG: --port="10250" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584528 4711 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584530 4711 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0581b9c4c2296ac53" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584533 4711 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584536 4711 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584539 4711 flags.go:64] FLAG: --register-node="true" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584542 4711 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584545 4711 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584548 4711 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584551 4711 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584556 4711 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584560 4711 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584563 4711 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584566 4711 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584568 4711 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584571 4711 flags.go:64] FLAG: --runonce="false" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584574 4711 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584577 4711 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584580 4711 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:01:36.592546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584583 4711 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584586 4711 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584589 4711 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584592 4711 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584596 4711 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584598 4711 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584601 4711 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584604 4711 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584607 4711 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584610 4711 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584612 4711 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584615 4711 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584620 4711 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584623 4711 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584626 4711 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584630 4711 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584633 4711 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584636 4711 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584639 4711 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584649 4711 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584652 4711 flags.go:64] FLAG: --v="2" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584656 4711 flags.go:64] FLAG: --version="false" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584660 4711 flags.go:64] FLAG: --vmodule="" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584667 4711 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.584670 4711 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:01:36.593186 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584792 4711 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584796 4711 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584798 4711 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584801 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584804 4711 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584806 4711 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584809 4711 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584811 4711 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584814 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584816 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584819 4711 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584822 4711 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584825 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584827 4711 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584830 4711 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584833 4711 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584837 4711 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584840 4711 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:36.593783 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584843 4711 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584846 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584848 4711 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584851 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584854 4711 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584856 4711 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584859 4711 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584861 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584864 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584867 4711 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584869 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584872 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584876 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584879 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584881 4711 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584884 4711 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584886 4711 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584889 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584891 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:36.594245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584894 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584896 4711 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584899 4711 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584901 4711 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584904 4711 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584906 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584909 4711 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584912 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584915 4711 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584917 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584919 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584922 4711 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584925 4711 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584927 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584930 4711 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584932 4711 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584934 4711 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584937 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584939 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584942 4711 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:36.594717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584945 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584947 4711 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584950 4711 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584952 4711 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584955 4711 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584958 4711 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584961 4711 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584964 4711 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584966 4711 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584969 4711 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584971 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584974 4711 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584977 4711 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584979 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584982 4711 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584984 4711 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584987 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584989 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584992 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584995 4711 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:36.595247 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.584998 4711 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585000 4711 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585003 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585005 4711 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585008 4711 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585010 4711 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585013 4711 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585015 4711 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585018 4711 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585021 4711 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585025 4711 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585028 4711 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585031 4711 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585034 4711 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585036 4711 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585039 4711 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585041 4711 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585045 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585048 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:36.596010 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585051 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585053 4711 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585056 4711 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585058 4711 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585061 4711 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585063 4711 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585066 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585068 4711 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585071 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585073 4711 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.585097 4711 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585131 4711 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585135 4711 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585139 4711 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585142 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585145 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:36.597293 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585147 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585150 4711 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585153 4711 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585155 4711 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585158 4711 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585161 4711 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585163 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585166 4711 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585168 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585170 4711 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585173 4711 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585176 4711 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585178 4711 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585181 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585183 4711 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585187 4711 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585190 4711 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585192 4711 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585195 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585197 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:36.597809 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585200 4711 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585203 4711 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585205 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585208 4711 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585211 4711 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585215 4711 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585219 4711 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585222 4711 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585226 4711 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585229 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585233 4711 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585236 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585238 4711 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585241 4711 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585244 4711 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585247 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585250 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585252 4711 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585255 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:36.598335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585257 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585259 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585262 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585264 4711 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585267 4711 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585269 4711 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585272 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585274 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585278 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585280 4711 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585283 4711 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585285 4711 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585288 4711 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585290 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585292 4711 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585295 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585298 4711 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585301 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585303 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585305 4711 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:36.598962 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585308 4711 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585310 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585313 4711 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585315 4711 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585318 4711 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585321 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585323 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585325 4711 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585329 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585331 4711 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585334 4711 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585336 4711 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585339 4711 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585341 4711 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585343 4711 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585346 4711 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585349 4711 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585351 4711 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585353 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585356 4711 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:36.599779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585360 4711 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585362 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585365 4711 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585368 4711 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585370 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585373 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585375 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585378 4711 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585380 4711 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585383 4711 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585385 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585388 4711 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585391 4711 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585393 4711 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585396 4711 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585398 4711 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585401 4711 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585404 4711 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585409 4711 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:36.600301 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585412 4711 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585415 4711 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.585418 4711 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.590150 4711 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.590166 4711 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.590185 4711 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.590190 4711 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590243 4711 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590248 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590252 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590255 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590257 4711 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590260 4711 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590262 4711 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590265 4711 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590267 4711 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590270 4711 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590273 4711 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590275 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:36.600913 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590278 4711 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590281 4711 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590284 4711 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590287 4711 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590289 4711 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590292 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590294 4711 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590297 4711 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590299 4711 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590302 4711 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590304 4711 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590307 4711 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590309 4711 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590312 4711 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590315 4711 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590317 4711 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590320 4711 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590323 4711 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590326 4711 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590328 4711 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:36.601526 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590330 4711 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590333 4711 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590335 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590338 4711 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590341 4711 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590343 4711 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590345 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590348 4711 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590351 4711 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590353 4711 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590355 4711 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590358 4711 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590360 4711 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590363 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590366 4711 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590369 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590371 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590374 4711 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590376 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:36.602035 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590379 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590381 4711 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590384 4711 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590386 4711 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590389 4711 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590392 4711 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590394 4711 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590397 4711 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590399 4711 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590401 4711 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590404 4711 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590407 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590409 4711 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590412 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590414 4711 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590418 4711 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590421 4711 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590423 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590426 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590428 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:36.602515 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590431 4711 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590433 4711 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590435 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590438 4711 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590441 4711 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590462 4711 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590464 4711 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590467 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590471 4711 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590474 4711 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590477 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590480 4711 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590483 4711 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590485 4711 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590488 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590490 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590493 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590496 4711 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590498 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:36.603014 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590501 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590503 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590506 4711 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590508 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590510 4711 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590513 4711 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590516 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590518 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590521 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590523 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590526 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590528 4711 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590531 4711 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590534 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590536 4711 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590538 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:36.603541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.590566 4711 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590722 4711 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590728 4711 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590731 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590733 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590736 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590739 4711 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590741 4711 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590744 4711 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590746 4711 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590748 4711 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590751 4711 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590754 4711 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590756 4711 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590758 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590761 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590764 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590766 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590768 4711 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590771 4711 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:36.603941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590773 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590775 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590784 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590787 4711 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590789 4711 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590791 4711 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590794 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590797 4711 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590799 4711 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590802 4711 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590804 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590806 4711 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590809 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590811 4711 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590814 4711 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590816 4711 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590818 4711 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590820 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590823 4711 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590825 4711 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:36.604409 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590827 4711 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590830 4711 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590832 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590835 4711 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590837 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590840 4711 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590842 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590845 4711 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590847 4711 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590850 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590852 4711 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590854 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590857 4711 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590860 4711 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590862 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590870 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590873 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590875 4711 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590878 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:36.604905 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590880 4711 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590883 4711 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590885 4711 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590888 4711 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590890 4711 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590893 4711 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590895 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590899 4711 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590902 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590904 4711 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590907 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590910 4711 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590912 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590915 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590917 4711 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590919 4711 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590922 4711 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590930 4711 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590933 4711 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590935 4711 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:36.605351 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590937 4711 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590940 4711 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590942 4711 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590945 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590947 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590949 4711 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590952 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590954 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590957 4711 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590965 4711 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590967 4711 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590970 4711 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590972 4711 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590974 4711 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590977 4711 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590979 4711 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590982 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590984 4711 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590987 4711 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590989 4711 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:36.605869 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590991 4711 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590994 4711 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.590998 4711 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591002 4711 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591005 4711 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591009 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591011 4711 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591015 4711 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.591037 4711 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591073 4711 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591078 4711 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591080 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591084 4711 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591087 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591090 4711 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:01:36.606438 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591093 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591095 4711 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591097 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591100 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591102 4711 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591104 4711 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591107 4711 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591115 4711 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591118 4711 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591120 4711 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591123 4711 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591125 4711 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591128 4711 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591130 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591135 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591137 4711 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591140 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591142 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591145 4711 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591147 4711 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:01:36.606918 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591149 4711 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591152 4711 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591154 4711 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591157 4711 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591159 4711 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591162 4711 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591164 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591166 4711 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591169 4711 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591171 4711 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591174 4711 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591176 4711 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591180 4711 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591184 4711 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591188 4711 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591190 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591193 4711 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591195 4711 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591198 4711 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:01:36.607505 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591202 4711 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591210 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591213 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591215 4711 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591218 4711 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591220 4711 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591223 4711 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591227 4711 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591229 4711 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591232 4711 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591234 4711 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591237 4711 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591239 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591242 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591244 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591247 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591249 4711 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591252 4711 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591254 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:01:36.608104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591257 4711 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591260 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591262 4711 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591264 4711 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591267 4711 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591270 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591272 4711 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591275 4711 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591277 4711 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591279 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591282 4711 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591284 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591287 4711 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591290 4711 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591293 4711 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591300 4711 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591303 4711 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591306 4711 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591308 4711 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591311 4711 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:01:36.608741 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591316 4711 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591319 4711 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591322 4711 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591324 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591327 4711 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591330 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591332 4711 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591335 4711 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591338 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591340 4711 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591342 4711 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591345 4711 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591347 4711 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591350 4711 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591352 4711 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591355 4711 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591357 4711 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591360 4711 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591364 4711 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:01:36.609493 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591366 4711 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591368 4711 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:36.591371 4711 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.591425 4711 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.592400 4711 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.592465 4711 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.592668 4711 server.go:1014] "Starting client certificate rotation" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.592796 4711 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.592839 4711 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:55:23 +0000 UTC" deadline="2028-07-23 09:09:35.500811464 +0000 UTC" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.592908 4711 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15061h7m58.907905734s" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.593204 4711 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.594830 4711 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.597920 4711 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.600694 4711 log.go:25] "Validated CRI v1 image API" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.601399 4711 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:01:36.609948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.603054 4711 fs.go:135] Filesystem UUIDs: map[4f7bb23f-23d6-4ee9-adad-5d1a0717478f:/dev/nvme0n1p4 5b4d2772-1258-45af-b013-31ec7ffdea05:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Nov 03 20:01:36.610324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.603068 4711 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a700b06825ad6ae433ac671c5b872229d064607749f41d72c7a26353abcd058/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a700b06825ad6ae433ac671c5b872229d064607749f41d72c7a26353abcd058/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f300d4c9fab06761d5bc3c414c322c1070d0277d3b98e626d704a9233539a89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f300d4c9fab06761d5bc3c414c322c1070d0277d3b98e626d704a9233539a89/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24033f77e98e2b98387bbff636ca30f4fbcb5c270f4e650dfba84d475b2b87e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24033f77e98e2b98387bbff636ca30f4fbcb5c270f4e650dfba84d475b2b87e9/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c50c65d22129c93e1ee1764729642e2057307959ce8bc58c62c0c2bf0044439/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c50c65d22129c93e1ee1764729642e2057307959ce8bc58c62c0c2bf0044439/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4584496f79d6ca178888660af44772fa061029d0b91271d4ce816f43c070b4ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4584496f79d6ca178888660af44772fa061029d0b91271d4ce816f43c070b4ff/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46d08c0c52558027466795f3a7a09ab2a28e10d941d2fa1bff4dbc7cff288ea8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46d08c0c52558027466795f3a7a09ab2a28e10d941d2fa1bff4dbc7cff288ea8/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/622e2d08eee35041ae23385941f7c82d5a4d4a7847d17ff0604944ba4cdf0258/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/622e2d08eee35041ae23385941f7c82d5a4d4a7847d17ff0604944ba4cdf0258/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7602a9a2d9313ba1c9e42dbc0b83494574da98dad0388c9b86ec067c38125392/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7602a9a2d9313ba1c9e42dbc0b83494574da98dad0388c9b86ec067c38125392/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/847954b04a80309354f4c24463571bb8ede786c010c70c68fe5680c9270de4ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/847954b04a80309354f4c24463571bb8ede786c010c70c68fe5680c9270de4ac/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/886cf21afb9cf752cdb9b52f66d860a9b8896e7869d1ba8fdcfe2209f99470ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/886cf21afb9cf752cdb9b52f66d860a9b8896e7869d1ba8fdcfe2209f99470ac/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88e618455f8f22e472094ce96e7ddd4ffb1c16231f006da80f767e0977d456fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88e618455f8f22e472094ce96e7ddd4ffb1c16231f006da80f767e0977d456fe/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa3f18d1385fd1555d70858586142eee2f8e55186c0161a49f9b096aecb56b3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa3f18d1385fd1555d70858586142eee2f8e55186c0161a49f9b096aecb56b3c/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af3faa9407b7346ff5c2f1886300b5fdb7853496a98153bfc21c30fa280adda6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af3faa9407b7346ff5c2f1886300b5fdb7853496a98153bfc21c30fa280adda6/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bb03c65f0709ceac8b7ab5131b0c4df7fb28c2cc6c669d49b14aed783cfcb2b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bb03c65f0709ceac8b7ab5131b0c4df7fb28c2cc6c669d49b14aed783cfcb2b6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d37bd4fe64f280a4e42d76022d6f13a2367dabd5d16c77277c3161b8ceaf480a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d37bd4fe64f280a4e42d76022d6f13a2367dabd5d16c77277c3161b8ceaf480a/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d486cf1ec6c7166adcdee9f01f30056eaf1e40ea2510484e4da6b9a37905ec6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d486cf1ec6c7166adcdee9f01f30056eaf1e40ea2510484e4da6b9a37905ec6c/userdata/shm major:0 minor:251 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dec9ccdd820d474ec01255a4bece78c0cf893625b5b1f56a7b69ac0e856ff216/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dec9ccdd820d474ec01255a4bece78c0cf893625b5b1f56a7b69ac0e856ff216/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f0cba8e705823b06d02bad90ef7f7806afaa9ef45e6d46342e1ceaea064538fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f0cba8e705823b06d02bad90ef7f7806afaa9ef45e6d46342e1ceaea064538fe/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9c98cbda157f34909384446fce81fe2770c3486fb529684caadcc23623a97f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9c98cbda157f34909384446fce81fe2770c3486fb529684caadcc23623a97f4/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/042f2f33-8a62-4464-bdd8-1a54ed18272d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/042f2f33-8a62-4464-bdd8-1a54ed18272d/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e17c916-ecd2-4ccd-bfeb-451380f47d30/volumes/kubernetes.io~projected/kube-api-access-4svzp:{mountpoint:/var/lib/kubelet/pods/1e17c916-ecd2-4ccd-bfeb-451380f47d30/volumes/kubernetes.io~projected/kube-api-access-4svzp major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ffacfcf-65b1-4809-8930-3c2952a90fc8/volumes/kubernetes.io~projected/kube-api-access-4bm4n:{mountpoint:/var/lib/kubelet/pods/2ffacfcf-65b1-4809-8930-3c2952a90fc8/volumes/kubernetes.io~projected/kube-api-access-4bm4n major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ffacfcf-65b1-4809-8930-3c2952a90fc8/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/2ffacfcf-65b1-4809-8930-3c2952a90fc8/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ffacfcf-65b1-4809-8930-3c2952a90fc8/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/2ffacfcf-65b1-4809-8930-3c2952a90fc8/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43579876-2c2b-4730-bd82-717f9b116bab/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/43579876-2c2b-4730-bd82-717f9b116bab/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43579876-2c2b-4730-bd82-717f9b116bab/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/43579876-2c2b-4730-bd82-717f9b116bab/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43579876-2c2b-4730-bd82-717f9b116bab/volumes/kubernetes.io~projected/kube-api-access-hj7dj:{mountpoint:/var/lib/kubelet/pods/43579876-2c2b-4730-bd82-717f9b116bab/volumes/kubernetes.io~projected/kube-api-access-hj7dj major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e75b735-0229-429d-927f-5683bcd373d6/volumes/kubernetes.io~projected/kube-api-access-tr474:{mountpoint:/var/lib/kubelet/pods/7e75b735-0229-429d-927f-5683bcd373d6/volumes/kubernetes.io~projected/kube-api-access-tr474 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e75b735-0229-429d-927f-5683bcd373d6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/7e75b735-0229-429d-927f-5683bcd373d6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:133 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91a2c232-0afd-43ed-a457-c977e4961416/volumes/kubernetes.io~projected/kube-api-access-djpvm:{mountpoint:/var/lib/kubelet/pods/91a2c232-0afd-43ed-a457-c977e4961416/volumes/kubernetes.io~projected/kube-api-access-djpvm major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91a2c232-0afd-43ed-a457-c977e4961416/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/91a2c232-0afd-43ed-a457-c977e4961416/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92b5617d-95cb-44c2-84b5-a73a50c2d749/volumes/kubernetes.io~projected/kube-api-access-ptlpz:{mountpoint:/var/lib/kubelet/pods/92b5617d-95cb-44c2-84b5-a73a50c2d749/volumes/kubernetes.io~projected/kube-api-access-ptlpz major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92b5617d-95cb-44c2-84b5-a73a50c2d749/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/92b5617d-95cb-44c2-84b5-a73a50c2d749/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af6745a-681c-44b6-861c-310df033cfe9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9af6745a-681c-44b6-861c-310df033cfe9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af6745a-681c-44b6-861c-310df033cfe9/volumes/kubernetes.io~projected/kube-api-access-7g8w7:{mountpoint:/var/lib/kubelet/pods/9af6745a-681c-44b6-861c-310df033cfe9/volumes/kubernetes.io~projected/kube-api-access-7g8w7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9af6745a-681c-44b6-861c-310df033cfe9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9af6745a-681c-44b6-861c-310df033cfe9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b/volumes/kubernetes.io~projected/kube-api-access-wsfgx:{mountpoint:/var/lib/kubelet/pods/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b/volumes/kubernetes.io~projected/kube-api-access-wsfgx major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a/volumes/kubernetes.io~projected/kube-api-access-7gb7w:{mountpoint:/var/lib/kubelet/pods/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a/volumes/kubernetes.io~projected/kube-api-access-7gb7w major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3f23518-97e0-4f3f-b939-d76052dc1339/volumes/kubernetes.io~projected/kube-api-access-sjvms:{mountpoint:/var/lib/kubelet/pods/a3f23518-97e0-4f3f-b939-d76052dc1339/volumes/kubernetes.io~projected/kube-api-access-sjvms major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9998949-10c8-4511-857c-dfad4001ab59/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a9998949-10c8-4511-857c-dfad4001ab59/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/baba1acd-b516-4149-a337-d56b9e74b36e/volumes/kubernetes.io~projected/kube-api-access-dz86h:{mountpoint:/var/lib/kubelet/pods/baba1acd-b516-4149-a337-d56b9e74b36e/volumes/kubernetes.io~projected/kube-api-access-dz86h major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd19fe74-5279-4b82-90e2-37bee00f7ccc/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/cd19fe74-5279-4b82-90e2-37bee00f7ccc/volumes/kubernetes.io~secret/tls-certificates major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d938e951-475c-47d6-8d73-f17da86400ad/volumes/kubernetes.io~projected/kube-api-access-dsj6v:{mountpoint:/var/lib/kubelet/pods/d938e951-475c-47d6-8d73-f17da86400ad/volumes/kubernetes.io~projected/kube-api-access-dsj6v major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969/volumes/kubernetes.io~projected/kube-api-access-ks7jx:{mountpoint:/var/lib/kubelet/pods/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969/volumes/kubernetes.io~projected/kube-api-access-ks7jx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969/volumes/kubernetes.io~secret/metrics-certs major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f/volumes/kubernetes.io~projected/kube-api-access-rxr52:{mountpoint:/var/lib/kubelet/pods/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f/volumes/kubernetes.io~projected/kube-api-access-rxr52 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbdb17a3-3ead-4c90-9922-31fbae907537/volumes/kubernetes.io~projected/kube-api-access-9flsn:{mountpoint:/var/lib/kubelet/pods/fbdb17a3-3ead-4c90-9922-31fbae907537/volumes/kubernetes.io~projected/kube-api-access-9flsn major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ce65b7e5845a3174e116dc3f38516e06e485e8fd73b6cd9b64efc9d325608337/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2b6f4ff09b06d8402118d059a39142719ba5afa24bb18c2fadc896ae4b613523/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/330642db4535bd0b99bce72f35efd2aec9462eb152f1d8639d68afa48e0d381e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/68ddec9c82cb4e65fff0f81de970ac3a6f6dd2b7e042e688d0982242fe737f2f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/6a453ecf6a227675e0f113c1403493e6bc48697f7093798993b094d330fdba0a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/fac0c4d71511d907f6a34d951dace326c2a06866be014c5ddbb5f4795bad1b72/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/9fd643ad40b458543941f8b186a855e5cf1a4a67e30179b5d01f304a6423519e/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9c0800f5c8a29ed73482ea0ff5192c933cf1dbb8af4c7dc4f37d0a4f35031839/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f89e3ad9e7aeb05742f8a8f5852a8e5f36f7555cef733ae39baf1cf1cdeb989b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/484ac28ac1275965e98ca632cf494ee1eca29a5e8f3d4bb27d914bb09b33a0e1/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/8fb4d7a25870f2b6dc57fe5be61abcd234a039455b0c317623de57545b525945/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/adf4b32ecd4653b43964e95f58d7f7df255664bd24415fcdf799ac9dddf62e41/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8568ee03dc6b5cc424ac2e2ea6c89a1e59d9e97231239aa252d8458eaba4848a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/ceec851d86ced75558057af0bb91e1bdac8fa67168b0e2d3683f88d1546573b5/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/73e00b583fcb41a3960caff765b035bd902b7f71ac642bed6838022336086d18/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/5bcf9875dc0922e813396fa0794a82f777fd1a415d53c05c7bf4d7851c1fb7ad/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/141ef681ae9373cfa253e53c00f58ccbb4c218dee72063d5a4670b083d28eb51/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/618905fe0b2e024144418adf0f1dbc87efa102a89786164510beb675f5a9b2cf/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/bd0cf89b64cee065d6a293b70bf6dce6c76fe756cc32bbf59c0036393945ede5/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/48752e877754bbb9ad78300f8364fd2aefcebbb6df44fcf2454d6d1b5cb4f055/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/e38c135b2fa3bb34ac71c9adceb19700e418c9766d1314a784bf0c9344a59898/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/3c1484958cc313a69521147c4149ea74e06dbb4599682d0550880fb71c98b6af/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/db040af9a864d61cee826c0c28c4f38b1c7fb19d9c2a0c105389fe3ceefa1b0e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/1022ff4ad8839e052f50bed678db2447e2be781e433b9b1fa463cdbc3bbff3bc/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/e9c52511c3c624b5459b02341f688b5a1204afd2e192917dd9fb6bdd38220efa/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/3b3572926ff3f5c5ea3e0f554734ed01b48bc26e4834224fc73569fb0c8c42d8/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/4cc1ac2564678f8e94ad62671b071297183b25df5c3503d744e26223aafa3124/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/b6162810859b73dd152dd49070883438b81eb6fe0d56bfa0748f55899a91d861/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/15272189d3e908f29914d3efa5ecb6a1989191cb9a736f282f622bbbc6d3ee2f/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/b296d5d00f7d84c8b271174460c012faa13bc7b41ea62c0c9fb9b2af3a4c92c9/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/0a6844066e9a708fc39a1f4ff4ab293e302bf7d18162c47dbc516befbb2ca8ed/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/af0308189759bd5d3f2dee3898478df5810d8746c365e0f2ad01d27cdb6a3f30/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/d8a6ace4294c9ba75459afb98f2270eac3782a0cff1131c5135d19050a134da4/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/99d945474ff278dc510d7a272d4dea1aac9a28e79ea93228a13afe58cbcc1c81/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/71a00c0b4563b72fe1014b9c6ce2dfa3438cb3eba160c000853d3ddabc0c00fd/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/f3ba9567409a63bd1632f5f67024e162f8487f2189295fdabedc65a574012d54/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/355232ee59072e715cebdab039473b8a5abc036a68aa2e3297141f0066ba20fc/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/d63dbac7a056305e5c5ba55f0432b32695aacf37825d420a6b7c0cc73a0e45a1/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/784ca5c96b13879f1e8167893b4529819107d525279fa7fcf49a52420911640f/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/09ff9f5c002272ead9f20efb758ff494eb6d62a43427d5a004b15bc10efb1476/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/a0e0bcec3bd9c3325705f49fa098567f70da2d4335a3819dc5c8810c3ed77948/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/8b1508fe258cf94cba3263a7ccc249900b6a00144f2173794c81a3e354075e57/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/c6f1efac21fe1e667cfa94313b5d7d42472e5400abbc05085021507ca95912a3/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/74ad9b6c13f4e1d1afcf3d42445e563573d8831b737892786d5a08b8c1e0b2d7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/acef8a53fac29859d3743d5a6af696697d067ec9842b54e4bcc302089347eade/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/33d9de0fe365bee8db50828ad03b9e1583cbc66ba95a1c2de550ba4abdba4a6d/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/57fa892fa8658a582507b11281bffaf161d53f87eec1a44cc1ffff0815ca3f8e/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/7be51efbea4485ec3facc136ed3131f34be66f679db28619967aebc074ade106/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b9aa6387fb426a447391c9caf22d14147dbcaf635956104ecfb3152c6e764c19/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/afed830466a47849295822a369d621aea0f27283efcbc3ddac0416955b9473ed/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/2165974bae9927435fa348d659abfa928466882bc60df448f14f1fad27417932/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a84f850e672694954e50781f4f9b579972942bbef96edeb3575ae0614ddae8b2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/043d1a4d8b8b23184715f690a4c72f421a73272e01ce1fa00097bc1ada20ce2a/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 20:01:36.610554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.609742 4711 manager.go:217] Machine: {Timestamp:2026-11-03 20:01:36.609248665 +0000 UTC m=+0.059354701 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3202343 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec283fabaf69111b6b6d2761d33fe7b6 SystemUUID:ec283fab-af69-111b-6b6d-2761d33fe7b6 BootID:5c2453d4-616a-4396-8552-391b411092a0 Filesystems:[{Device:/var/lib/kubelet/pods/baba1acd-b516-4149-a337-d56b9e74b36e/volumes/kubernetes.io~projected/kube-api-access-dz86h DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af3faa9407b7346ff5c2f1886300b5fdb7853496a98153bfc21c30fa280adda6/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7e75b735-0229-429d-927f-5683bcd373d6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:133 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cd19fe74-5279-4b82-90e2-37bee00f7ccc/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92b5617d-95cb-44c2-84b5-a73a50c2d749/volumes/kubernetes.io~projected/kube-api-access-ptlpz DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bb03c65f0709ceac8b7ab5131b0c4df7fb28c2cc6c669d49b14aed783cfcb2b6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f/volumes/kubernetes.io~projected/kube-api-access-rxr52 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7602a9a2d9313ba1c9e42dbc0b83494574da98dad0388c9b86ec067c38125392/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d37bd4fe64f280a4e42d76022d6f13a2367dabd5d16c77277c3161b8ceaf480a/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f0cba8e705823b06d02bad90ef7f7806afaa9ef45e6d46342e1ceaea064538fe/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24033f77e98e2b98387bbff636ca30f4fbcb5c270f4e650dfba84d475b2b87e9/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a9998949-10c8-4511-857c-dfad4001ab59/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:241 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969/volumes/kubernetes.io~projected/kube-api-access-ks7jx DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e75b735-0229-429d-927f-5683bcd373d6/volumes/kubernetes.io~projected/kube-api-access-tr474 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c50c65d22129c93e1ee1764729642e2057307959ce8bc58c62c0c2bf0044439/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/43579876-2c2b-4730-bd82-717f9b116bab/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9af6745a-681c-44b6-861c-310df033cfe9/volumes/kubernetes.io~projected/kube-api-access-7g8w7 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f300d4c9fab06761d5bc3c414c322c1070d0277d3b98e626d704a9233539a89/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d938e951-475c-47d6-8d73-f17da86400ad/volumes/kubernetes.io~projected/kube-api-access-dsj6v DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9af6745a-681c-44b6-861c-310df033cfe9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91a2c232-0afd-43ed-a457-c977e4961416/volumes/kubernetes.io~projected/kube-api-access-djpvm DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88e618455f8f22e472094ce96e7ddd4ffb1c16231f006da80f767e0977d456fe/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/847954b04a80309354f4c24463571bb8ede786c010c70c68fe5680c9270de4ac/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/886cf21afb9cf752cdb9b52f66d860a9b8896e7869d1ba8fdcfe2209f99470ac/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d486cf1ec6c7166adcdee9f01f30056eaf1e40ea2510484e4da6b9a37905ec6c/userdata/shm DeviceMajor:0 DeviceMinor:251 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91a2c232-0afd-43ed-a457-c977e4961416/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa3f18d1385fd1555d70858586142eee2f8e55186c0161a49f9b096aecb56b3c/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a/volumes/kubernetes.io~projected/kube-api-access-7gb7w DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/622e2d08eee35041ae23385941f7c82d5a4d4a7847d17ff0604944ba4cdf0258/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4584496f79d6ca178888660af44772fa061029d0b91271d4ce816f43c070b4ff/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b/volumes/kubernetes.io~projected/kube-api-access-wsfgx DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ffacfcf-65b1-4809-8930-3c2952a90fc8/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:268 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ffacfcf-65b1-4809-8930-3c2952a90fc8/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:273 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1e17c916-ecd2-4ccd-bfeb-451380f47d30/volumes/kubernetes.io~projected/kube-api-access-4svzp DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9af6745a-681c-44b6-861c-310df033cfe9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/43579876-2c2b-4730-bd82-717f9b116bab/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a3f23518-97e0-4f3f-b939-d76052dc1339/volumes/kubernetes.io~projected/kube-api-access-sjvms DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9c98cbda157f34909384446fce81fe2770c3486fb529684caadcc23623a97f4/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ffacfcf-65b1-4809-8930-3c2952a90fc8/volumes/kubernetes.io~projected/kube-api-access-4bm4n DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46d08c0c52558027466795f3a7a09ab2a28e10d941d2fa1bff4dbc7cff288ea8/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43579876-2c2b-4730-bd82-717f9b116bab/volumes/kubernetes.io~projected/kube-api-access-hj7dj DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a700b06825ad6ae433ac671c5b872229d064607749f41d72c7a26353abcd058/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/042f2f33-8a62-4464-bdd8-1a54ed18272d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92b5617d-95cb-44c2-84b5-a73a50c2d749/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:200 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dec9ccdd820d474ec01255a4bece78c0cf893625b5b1f56a7b69ac0e856ff216/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbdb17a3-3ead-4c90-9922-31fbae907537/volumes/kubernetes.io~projected/kube-api-access-9flsn DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a700b06825ad6a MacAddress:46:ff:58:ce:39:92 Speed:10000 Mtu:8901} {Name:0f300d4c9fab067 MacAddress:ba:76:18:0d:c3:34 Speed:10000 Mtu:8901} {Name:2c50c65d22129c9 MacAddress:96:a4:c8:c4:42:b5 Speed:10000 Mtu:8901} {Name:7602a9a2d9313ba MacAddress:06:ac:e2:22:c1:83 Speed:10000 Mtu:8901} {Name:847954b04a80309 MacAddress:ae:f5:f0:6c:6b:a4 Speed:10000 Mtu:8901} {Name:886cf21afb9cf75 MacAddress:de:32:cd:bb:88:3e Speed:10000 Mtu:8901} {Name:aa3f18d1385fd15 MacAddress:f2:00:11:73:bd:44 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d1:24:85:ad Speed:0 Mtu:9001} {Name:br-int MacAddress:76:7d:eb:b0:54:88 Speed:0 Mtu:8901} {Name:d486cf1ec6c7166 MacAddress:5a:47:4a:ac:78:df Speed:10000 Mtu:8901} {Name:dec9ccdd820d474 MacAddress:32:db:9a:bf:7d:f8 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d1:24:85:ad Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:aa:32:b8:a9:83:6f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ba:9a:56:b0:41:fc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:01:36.610554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610015 4711 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:01:36.610554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610085 4711 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:01:36.610554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610413 4711 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:01:36.610766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610431 4711 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-5.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:01:36.610766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610578 4711 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:01:36.610766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610586 4711 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:01:36.610766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610598 4711 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:01:36.610766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610616 4711 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:01:36.610766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610689 4711 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:01:36.610939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.610841 4711 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:01:36.611097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.611086 4711 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:01:36.611125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.611121 4711 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:01:36.611150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.611134 4711 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:01:36.611150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.611144 4711 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:01:36.611205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.611153 4711 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:01:36.611205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.611160 4711 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:01:36.611813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.611800 4711 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:01:36.611859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.611815 4711 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:01:36.612318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.612298 4711 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:01:36.624378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.624346 4711 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:01:36.624378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.624363 4711 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:01:36.624554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.624515 4711 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 20:01:36.624823 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.624808 4711 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:01:36.624902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.624856 4711 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:01:36.624985 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.624973 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:01:36.625046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.624993 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:01:36.625046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625004 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:01:36.625046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625013 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:01:36.625046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625029 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:01:36.625046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625039 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:01:36.625366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625049 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:01:36.625366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625058 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:01:36.625366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625070 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:01:36.625366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625080 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:01:36.625366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625093 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:01:36.625366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625108 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:01:36.625366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625142 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:01:36.625366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.625152 4711 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:01:36.629245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.629231 4711 server.go:1263] "Started kubelet" Nov 03 20:01:36.629346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.629312 4711 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:01:36.629471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.629378 4711 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:01:36.629541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.629489 4711 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:01:36.629761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.629745 4711 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:01:36.629893 ip-10-0-142-5 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:01:36.630834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.630808 4711 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:01:36.630932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.630860 4711 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:01:36.632661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.632615 4711 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:01:36.633334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.632740 4711 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:01:36.633334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.632792 4711 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:01:36.633334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.632857 4711 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:01:36.633334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.632790 4711 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:55:25 +0000 UTC" deadline="2028-04-03 09:24:04.026215809 +0000 UTC" Nov 03 20:01:36.633334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.632920 4711 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12397h22m27.393299758s" Nov 03 20:01:36.633334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.632941 4711 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.634950 4711 factory.go:55] Registering systemd factory Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.634972 4711 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637252 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e17c916-ecd2-4ccd-bfeb-451380f47d30" volumeName="kubernetes.io/projected/1e17c916-ecd2-4ccd-bfeb-451380f47d30-kube-api-access-4svzp" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637300 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ffacfcf-65b1-4809-8930-3c2952a90fc8" volumeName="kubernetes.io/configmap/2ffacfcf-65b1-4809-8930-3c2952a90fc8-metrics-client-ca" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637316 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43579876-2c2b-4730-bd82-717f9b116bab" volumeName="kubernetes.io/projected/43579876-2c2b-4730-bd82-717f9b116bab-kube-api-access-hj7dj" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637333 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af6745a-681c-44b6-861c-310df033cfe9" volumeName="kubernetes.io/projected/9af6745a-681c-44b6-861c-310df033cfe9-kube-api-access-7g8w7" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637351 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f23518-97e0-4f3f-b939-d76052dc1339" volumeName="kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-daemon-config" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637365 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd19fe74-5279-4b82-90e2-37bee00f7ccc" volumeName="kubernetes.io/secret/cd19fe74-5279-4b82-90e2-37bee00f7ccc-tls-certificates" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637383 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43579876-2c2b-4730-bd82-717f9b116bab" volumeName="kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-tmp" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637397 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e75b735-0229-429d-927f-5683bcd373d6" volumeName="kubernetes.io/projected/7e75b735-0229-429d-927f-5683bcd373d6-kube-api-access-tr474" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637419 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f23518-97e0-4f3f-b939-d76052dc1339" volumeName="kubernetes.io/projected/a3f23518-97e0-4f3f-b939-d76052dc1339-kube-api-access-sjvms" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637431 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9998949-10c8-4511-857c-dfad4001ab59" volumeName="kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637459 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baba1acd-b516-4149-a337-d56b9e74b36e" volumeName="kubernetes.io/projected/baba1acd-b516-4149-a337-d56b9e74b36e-kube-api-access-dz86h" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637478 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbdb17a3-3ead-4c90-9922-31fbae907537" volumeName="kubernetes.io/empty-dir/fbdb17a3-3ead-4c90-9922-31fbae907537-tmp-dir" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637494 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="042f2f33-8a62-4464-bdd8-1a54ed18272d" volumeName="kubernetes.io/configmap/042f2f33-8a62-4464-bdd8-1a54ed18272d-konnectivity-ca" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637514 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ffacfcf-65b1-4809-8930-3c2952a90fc8" volumeName="kubernetes.io/projected/2ffacfcf-65b1-4809-8930-3c2952a90fc8-kube-api-access-4bm4n" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637535 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e75b735-0229-429d-927f-5683bcd373d6" volumeName="kubernetes.io/empty-dir/7e75b735-0229-429d-927f-5683bcd373d6-tmp-dir" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637554 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a2c232-0afd-43ed-a457-c977e4961416" volumeName="kubernetes.io/projected/91a2c232-0afd-43ed-a457-c977e4961416-kube-api-access-djpvm" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637569 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91a2c232-0afd-43ed-a457-c977e4961416" volumeName="kubernetes.io/secret/91a2c232-0afd-43ed-a457-c977e4961416-cert" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637584 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="baba1acd-b516-4149-a337-d56b9e74b36e" volumeName="kubernetes.io/configmap/baba1acd-b516-4149-a337-d56b9e74b36e-serviceca" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637602 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d938e951-475c-47d6-8d73-f17da86400ad" volumeName="kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637614 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbdb17a3-3ead-4c90-9922-31fbae907537" volumeName="kubernetes.io/projected/fbdb17a3-3ead-4c90-9922-31fbae907537-kube-api-access-9flsn" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637635 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ffacfcf-65b1-4809-8930-3c2952a90fc8" volumeName="kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637647 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af6745a-681c-44b6-861c-310df033cfe9" volumeName="kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-env-overrides" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637664 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af6745a-681c-44b6-861c-310df033cfe9" volumeName="kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-config" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637678 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af6745a-681c-44b6-861c-310df033cfe9" volumeName="kubernetes.io/secret/9af6745a-681c-44b6-861c-310df033cfe9-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637694 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" volumeName="kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637712 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="042f2f33-8a62-4464-bdd8-1a54ed18272d" volumeName="kubernetes.io/secret/042f2f33-8a62-4464-bdd8-1a54ed18272d-agent-certs" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637727 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43579876-2c2b-4730-bd82-717f9b116bab" volumeName="kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-etc-tuned" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637746 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e75b735-0229-429d-927f-5683bcd373d6" volumeName="kubernetes.io/configmap/7e75b735-0229-429d-927f-5683bcd373d6-config-volume" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637781 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92b5617d-95cb-44c2-84b5-a73a50c2d749" volumeName="kubernetes.io/projected/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-api-access-ptlpz" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637800 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d938e951-475c-47d6-8d73-f17da86400ad" volumeName="kubernetes.io/projected/d938e951-475c-47d6-8d73-f17da86400ad-kube-api-access-dsj6v" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637814 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ffacfcf-65b1-4809-8930-3c2952a90fc8" volumeName="kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-tls" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637833 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e75b735-0229-429d-927f-5683bcd373d6" volumeName="kubernetes.io/secret/7e75b735-0229-429d-927f-5683bcd373d6-metrics-tls" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637847 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9af6745a-681c-44b6-861c-310df033cfe9" volumeName="kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-script-lib" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637862 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b" volumeName="kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637881 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2cc4b09-f2b3-4eb1-93fa-41ab773f995a" volumeName="kubernetes.io/configmap/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-iptables-alerter-script" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637895 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e17c916-ecd2-4ccd-bfeb-451380f47d30" volumeName="kubernetes.io/empty-dir/1e17c916-ecd2-4ccd-bfeb-451380f47d30-tmp" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637913 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92b5617d-95cb-44c2-84b5-a73a50c2d749" volumeName="kubernetes.io/empty-dir/92b5617d-95cb-44c2-84b5-a73a50c2d749-data-volume" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637928 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92b5617d-95cb-44c2-84b5-a73a50c2d749" volumeName="kubernetes.io/secret/92b5617d-95cb-44c2-84b5-a73a50c2d749-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637942 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3f23518-97e0-4f3f-b939-d76052dc1339" volumeName="kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-cni-binary-copy" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637958 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d938e951-475c-47d6-8d73-f17da86400ad" volumeName="kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637969 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d938e951-475c-47d6-8d73-f17da86400ad" volumeName="kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-binary-copy" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637985 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f" volumeName="kubernetes.io/projected/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kube-api-access-rxr52" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.637997 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92b5617d-95cb-44c2-84b5-a73a50c2d749" volumeName="kubernetes.io/configmap/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638009 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2cc4b09-f2b3-4eb1-93fa-41ab773f995a" volumeName="kubernetes.io/projected/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-kube-api-access-7gb7w" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638029 4711 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969" volumeName="kubernetes.io/projected/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-kube-api-access-ks7jx" seLinuxMountContext="" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638041 4711 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638050 4711 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638314 4711 factory.go:153] Registering CRI-O factory Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638331 4711 factory.go:223] Registration of the crio container factory successfully Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638388 4711 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638404 4711 factory.go:103] Registering Raw factory Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638416 4711 manager.go:1199] Started watching for new ooms in manager Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.638715 4711 manager.go:319] Starting recovery of all containers Nov 03 20:01:36.641301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.641072 4711 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:01:36.647109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.646174 4711 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:01:36.648149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.648129 4711 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 20:01:36.652033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.651904 4711 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:01:36.655942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.655695 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:01:36.662264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.662246 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:01:36.666222 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.666206 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:01:36.671738 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.671402 4711 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:01:36.671827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.671743 4711 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:01:36.671827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.671768 4711 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:01:36.671924 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:01:36.671818 4711 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 20:01:36.673225 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.673204 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:01:36.683298 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.683283 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:01:36.683389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.683287 4711 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:01:36.683712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.683690 4711 generic.go:356] "Generic (PLEG): container finished" podID="9af6745a-681c-44b6-861c-310df033cfe9" containerID="2cf9cb15ac5be549bcf98d35578c64555626a3c32c181802373ff4ce8668e538" exitCode=1 Nov 03 20:01:36.698743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.698728 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:01:36.699507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.698754 4711 generic.go:356] "Generic (PLEG): container finished" podID="a3f23518-97e0-4f3f-b939-d76052dc1339" containerID="cb449686831d4ada2d76f5bbd5bdf3d457625402cd1a4a58bd868669757956a5" exitCode=1 Nov 03 20:01:36.701115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.701094 4711 generic.go:356] "Generic (PLEG): container finished" podID="62881be2bc7cb00291f2b39128d27e0c" containerID="eb3d1288bbad91b647d5c022ef77fe7786337e29c6bc6e416c8b7b99873e5103" exitCode=0 Nov 03 20:01:36.734554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.734518 4711 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="2bea7681c057f63ffac7b2821efe7fa7eea2c3ebcec56d94714840aea8ee8322" exitCode=0 Nov 03 20:01:36.734554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.734550 4711 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="a1e73fe6f50ceaf6d8c7f7a5ac62a249649f99dfbc033b5629ec93b4d86e3e68" exitCode=0 Nov 03 20:01:36.734554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.734559 4711 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="3023f0acb6b191a70f61fa8e533f1ac3af0e80058381f6750cedfd735b2aa7cf" exitCode=0 Nov 03 20:01:36.734792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.734565 4711 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="df482d9cea138e91f6418f61d1797b9e2fe60442228303047caa102366945921" exitCode=0 Nov 03 20:01:36.734792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.734570 4711 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="cb35a69aaee8fa08d2be748f4ba888d2623c914245259b0e7a331c67d71f00c4" exitCode=0 Nov 03 20:01:36.734792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.734576 4711 generic.go:356] "Generic (PLEG): container finished" podID="d938e951-475c-47d6-8d73-f17da86400ad" containerID="5a3619123f177e8fc287dd7732e4d6a891ab35d381c16356d85c61913fcd86bd" exitCode=0 Nov 03 20:01:36.739102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.739084 4711 manager.go:324] Recovery completed Nov 03 20:01:36.759553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759534 4711 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:01:36.759553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759548 4711 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:01:36.759732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759565 4711 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:01:36.759732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759711 4711 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 20:01:36.759732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759721 4711 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 20:01:36.759887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759739 4711 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 20:01:36.759887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759744 4711 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 20:01:36.759887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759749 4711 policy_none.go:50] "Start" Nov 03 20:01:36.759887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759757 4711 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:01:36.759887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759767 4711 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:01:36.759887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759879 4711 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 20:01:36.759887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759886 4711 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 20:01:36.759887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.759891 4711 policy_none.go:44] "Start" Nov 03 20:01:36.764802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.764271 4711 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:01:36.769435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.769421 4711 manager.go:341] "Starting Device Plugin manager" Nov 03 20:01:36.769556 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:01:36.769462 4711 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:01:36.769556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.769473 4711 server.go:87] "Starting device plugin registration server" Nov 03 20:01:36.769785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.769772 4711 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:01:36.769817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.769789 4711 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:01:36.769894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.769875 4711 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:01:36.770022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.769980 4711 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:01:36.770088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.770026 4711 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:01:36.770088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.770033 4711 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:01:36.770171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.770091 4711 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:01:36.770013848Z","UUID":"d6d4173f-b96c-44cf-881a-bae684dfba1d","Handler":null,"Name":"","Endpoint":""} Nov 03 20:01:36.771461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.771427 4711 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:01:36.771563 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.771476 4711 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:01:36.771704 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:01:36.771674 4711 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:01:36.772005 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.771965 4711 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal"] Nov 03 20:01:36.772668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.772625 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"23259ea2648a683e23b5758ee6ae68e7","Type":"ContainerStarted","Data":"25c22b402723be6ff25dee8536aea7f0adadff42eaf9a7b8a8184384787edd07"} Nov 03 20:01:36.772762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.772677 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"23259ea2648a683e23b5758ee6ae68e7","Type":"ContainerStarted","Data":"bb03c65f0709ceac8b7ab5131b0c4df7fb28c2cc6c669d49b14aed783cfcb2b6"} Nov 03 20:01:36.772762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.772727 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"62881be2bc7cb00291f2b39128d27e0c","Type":"ContainerStarted","Data":"5d55fd9976f79b0a19d7a0e989310f596dfe37db2fb20af7e58bd1f185369081"} Nov 03 20:01:36.772762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.772745 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"62881be2bc7cb00291f2b39128d27e0c","Type":"ContainerDied","Data":"eb3d1288bbad91b647d5c022ef77fe7786337e29c6bc6e416c8b7b99873e5103"} Nov 03 20:01:36.772762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.772764 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" event={"ID":"62881be2bc7cb00291f2b39128d27e0c","Type":"ContainerStarted","Data":"46d08c0c52558027466795f3a7a09ab2a28e10d941d2fa1bff4dbc7cff288ea8"} Nov 03 20:01:36.772983 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.772762 4711 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.772983 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.772879 4711 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.817381 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.817343 4711 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:01:36.817542 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:01:36.817414 4711 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.817542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.817480 4711 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:01:36.817542 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:01:36.817519 4711 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.849131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.849099 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.849131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.849128 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.849296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.849145 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23259ea2648a683e23b5758ee6ae68e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"23259ea2648a683e23b5758ee6ae68e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.870223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.870202 4711 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:01:36.872264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.872249 4711 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:01:36.872361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.872283 4711 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:01:36.872361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.872298 4711 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-5.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:01:36.872361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.872344 4711 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.892514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.892494 4711 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.892589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.892580 4711 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.949310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.949275 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23259ea2648a683e23b5758ee6ae68e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"23259ea2648a683e23b5758ee6ae68e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.949498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.949317 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.949498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.949340 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.949498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.949347 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/23259ea2648a683e23b5758ee6ae68e7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"23259ea2648a683e23b5758ee6ae68e7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.949498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.949384 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:36.949498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:36.949411 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/62881be2bc7cb00291f2b39128d27e0c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" (UID: \"62881be2bc7cb00291f2b39128d27e0c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:37.612027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.611979 4711 apiserver.go:52] "Watching apiserver" Nov 03 20:01:37.632594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.632566 4711 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:01:37.633067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.633044 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-tffb6","openshift-monitoring/node-exporter-q2skn","openshift-dns/node-resolver-wszxg","kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal","kube-system/global-pull-secret-syncer-bhl7x","openshift-multus/network-metrics-daemon-xf24w","openshift-cluster-node-tuning-operator/tuned-qc2tr","openshift-insights/insights-runtime-extractor-znlzm","openshift-multus/multus-56d9g","openshift-multus/multus-additional-cni-plugins-s7lv9","openshift-network-diagnostics/network-check-target-4xx8n","openshift-ovn-kubernetes/ovnkube-node-txw5p","openshift-console/downloads-6dd789c8b7-smhql","openshift-dns/dns-default-c8hzq","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr","openshift-image-registry/node-ca-5h7vn","openshift-ingress-canary/ingress-canary-7wjnx","kube-system/konnectivity-agent-8fpvs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal","openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5"] Nov 03 20:01:37.645644 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.645618 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.652648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.652622 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:01:37.652762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.652746 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:01:37.652830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.652753 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nc2g5\"" Nov 03 20:01:37.652830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.652808 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.652897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.652836 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.654309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.654289 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:01:37.654397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.654320 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.654397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.654344 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.654503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.654462 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:01:37.654566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.654507 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.654566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.654511 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:01:37.654566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.654534 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.654566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.654553 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.656096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.656083 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vbl6p\"" Nov 03 20:01:37.656152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.656097 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-whl79\"" Nov 03 20:01:37.656425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.656412 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 20:01:37.656490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.656414 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vlwgf\"" Nov 03 20:01:37.656490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.656484 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s55dl\"" Nov 03 20:01:37.656569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.656506 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:01:37.658274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.658033 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mszh\"" Nov 03 20:01:37.658274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.658045 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:01:37.658274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.658143 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.658274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.658154 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:01:37.658274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.658210 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.658274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.658219 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:01:37.659489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.659470 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.659542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.659508 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.659995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.659973 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.659995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.659990 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.660137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660047 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.660137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660081 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.660137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660097 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:01:37.660137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660124 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.660137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660136 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.660288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660150 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:01:37.660288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660151 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:01:37.660288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660159 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:01:37.660288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660178 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.660973 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660959 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:01:37.661040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660978 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-lrvk5\"" Nov 03 20:01:37.661040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.660999 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7vkxg\"" Nov 03 20:01:37.661040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.661031 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.661159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.661054 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:01:37.661692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.661677 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 20:01:37.661765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.661706 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:01:37.661765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.661720 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 20:01:37.661765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.661737 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:01:37.661765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.661748 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7gvxc\"" Nov 03 20:01:37.661889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.661791 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:01:37.663495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663481 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.663495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663489 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.663610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663521 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:01:37.663610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663547 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:01:37.663610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663558 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:01:37.663771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663746 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zfqpt\"" Nov 03 20:01:37.663830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663772 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:01:37.663830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663747 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8fc4f\"" Nov 03 20:01:37.663830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663754 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rnml9\"" Nov 03 20:01:37.663830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663788 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kmt87\"" Nov 03 20:01:37.663830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663798 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kcrwf\"" Nov 03 20:01:37.663996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663895 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gz7vs\"" Nov 03 20:01:37.663996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663942 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zc6h4\"" Nov 03 20:01:37.663996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.663963 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jrhnz\"" Nov 03 20:01:37.664148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.664131 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lxhjg\"" Nov 03 20:01:37.665315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.665302 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8ls9s\"" Nov 03 20:01:37.702512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.702481 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:01:37.702656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.702639 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:01:37.733177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.733141 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7wjnx_91a2c232-0afd-43ed-a457-c977e4961416/serve-healthcheck-canary/0.log" Nov 03 20:01:37.734100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.734078 4711 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:01:37.738062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.738043 4711 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:37.738223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.738206 4711 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:37.743943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.743923 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.744065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.744047 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:01:37.744128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.744089 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:01:37.753051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753027 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-kubelet\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.753120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753059 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr474\" (UniqueName: \"kubernetes.io/projected/7e75b735-0229-429d-927f-5683bcd373d6-kube-api-access-tr474\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:37.753120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753083 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ef7a29-08bf-4a3c-b815-a34698896df1-metrics-client-ca\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.753120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753112 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-sys\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.753205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753152 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-var-lib-kubelet\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.753205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753177 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-tmp\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.753266 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753242 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flsn\" (UniqueName: \"kubernetes.io/projected/fbdb17a3-3ead-4c90-9922-31fbae907537-kube-api-access-9flsn\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:01:37.753266 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753257 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-tmp\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.753389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753372 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-slash\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.753423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753402 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-ovn\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.753467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753424 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.753508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753478 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:37.753537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753517 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.753574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753554 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd19fe74-5279-4b82-90e2-37bee00f7ccc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hh4rb\" (UID: \"cd19fe74-5279-4b82-90e2-37bee00f7ccc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:37.753679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753664 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-bin\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.753788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753773 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-netd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.753829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753817 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-config\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.753862 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753848 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/baba1acd-b516-4149-a337-d56b9e74b36e-serviceca\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:01:37.754008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.753993 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e75b735-0229-429d-927f-5683bcd373d6-config-volume\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:37.754048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754005 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cd19fe74-5279-4b82-90e2-37bee00f7ccc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hh4rb\" (UID: \"cd19fe74-5279-4b82-90e2-37bee00f7ccc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:37.754082 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754070 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsj6v\" (UniqueName: \"kubernetes.io/projected/d938e951-475c-47d6-8d73-f17da86400ad-kube-api-access-dsj6v\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.754115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754087 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-config\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.754152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754119 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-conf\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.754216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754200 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e75b735-0229-429d-927f-5683bcd373d6-config-volume\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:37.754251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754228 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/baba1acd-b516-4149-a337-d56b9e74b36e-serviceca\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:01:37.754251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754234 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djpvm\" (UniqueName: \"kubernetes.io/projected/91a2c232-0afd-43ed-a457-c977e4961416-kube-api-access-djpvm\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:37.754312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754265 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/042f2f33-8a62-4464-bdd8-1a54ed18272d-konnectivity-ca\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:01:37.754312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754280 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e75b735-0229-429d-927f-5683bcd373d6-tmp-dir\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:37.754312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754296 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-system-cni-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.754401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754342 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91a2c232-0afd-43ed-a457-c977e4961416-cert\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:37.754401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754363 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e75b735-0229-429d-927f-5683bcd373d6-tmp-dir\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:37.754401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754369 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffacfcf-65b1-4809-8930-3c2952a90fc8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:37.754537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754401 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptlpz\" (UniqueName: \"kubernetes.io/projected/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-api-access-ptlpz\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.754537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754428 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-cnibin\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.754603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754530 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/042f2f33-8a62-4464-bdd8-1a54ed18272d-konnectivity-ca\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:01:37.754603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754542 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ffacfcf-65b1-4809-8930-3c2952a90fc8-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:37.754603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754579 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-registration-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.754700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754611 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91a2c232-0afd-43ed-a457-c977e4961416-cert\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:37.754700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754616 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-sys-fs\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.754700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754649 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-etc-tuned\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.754700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754688 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bm4n\" (UniqueName: \"kubernetes.io/projected/2ffacfcf-65b1-4809-8930-3c2952a90fc8-kube-api-access-4bm4n\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:37.754815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754723 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-node-log\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.754852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754833 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-log-socket\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.754885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754851 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-lib-modules\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.754885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754852 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43579876-2c2b-4730-bd82-717f9b116bab-etc-tuned\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.754885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754866 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90ef7a29-08bf-4a3c-b815-a34698896df1-sys\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.754975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754892 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlvm\" (UniqueName: \"kubernetes.io/projected/90ef7a29-08bf-4a3c-b815-a34698896df1-kube-api-access-6xlvm\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.754975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754916 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-kubelet\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.754975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754932 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.754975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754948 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-host-slash\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:01:37.754975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754971 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/042f2f33-8a62-4464-bdd8-1a54ed18272d-agent-certs\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:01:37.755125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.754988 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.755125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755029 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:37.755184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755119 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.755184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755155 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-env-overrides\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.755243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755182 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af6745a-681c-44b6-861c-310df033cfe9-ovn-node-metrics-cert\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.755243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755198 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.755243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755210 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz86h\" (UniqueName: \"kubernetes.io/projected/baba1acd-b516-4149-a337-d56b9e74b36e-kube-api-access-dz86h\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:01:37.755243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755237 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90ef7a29-08bf-4a3c-b815-a34698896df1-root\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.755390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755278 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-system-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.755390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755307 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-k8s-cni-cncf-io\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.755390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755308 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/042f2f33-8a62-4464-bdd8-1a54ed18272d-agent-certs\") pod \"konnectivity-agent-8fpvs\" (UID: \"042f2f33-8a62-4464-bdd8-1a54ed18272d\") " pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:01:37.755390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755361 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-metrics-certs\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:37.755390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755378 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.755626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755399 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-env-overrides\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.755626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755436 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.755626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755489 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7dj\" (UniqueName: \"kubernetes.io/projected/43579876-2c2b-4730-bd82-717f9b116bab-kube-api-access-hj7dj\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.755626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755556 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92b5617d-95cb-44c2-84b5-a73a50c2d749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.755626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755567 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9af6745a-681c-44b6-861c-310df033cfe9-ovn-node-metrics-cert\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.755626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755608 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-var-lib-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755657 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-device-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755713 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-modprobe-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755731 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755740 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755748 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-systemd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755756 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/92b5617d-95cb-44c2-84b5-a73a50c2d749-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755786 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e17c916-ecd2-4ccd-bfeb-451380f47d30-tmp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755809 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-bin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755826 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-multus\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.755859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755857 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gb7w\" (UniqueName: \"kubernetes.io/projected/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-kube-api-access-7gb7w\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755875 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-systemd-units\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755887 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e17c916-ecd2-4ccd-bfeb-451380f47d30-tmp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755893 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-etc-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755916 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-systemd\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755944 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.755966 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-socket-dir-parent\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756003 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-conf-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756024 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756042 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks7jx\" (UniqueName: \"kubernetes.io/projected/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-kube-api-access-ks7jx\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756161 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92b5617d-95cb-44c2-84b5-a73a50c2d749-data-volume\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756185 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8w7\" (UniqueName: \"kubernetes.io/projected/9af6745a-681c-44b6-861c-310df033cfe9-kube-api-access-7g8w7\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756203 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-textfile\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756224 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-tls\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756240 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-cnibin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756256 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/92b5617d-95cb-44c2-84b5-a73a50c2d749-data-volume\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756266 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvms\" (UniqueName: \"kubernetes.io/projected/a3f23518-97e0-4f3f-b939-d76052dc1339-kube-api-access-sjvms\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756264 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d938e951-475c-47d6-8d73-f17da86400ad-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.756303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756307 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-socket-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756336 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-etc-selinux\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756361 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-kubelet-config\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756384 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-host\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756416 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-run\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756488 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysconfig\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756515 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-kubernetes\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756540 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-wtmp\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756570 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756594 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756615 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-netns\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756658 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-hostroot\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756684 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-multus-certs\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756708 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-cni-binary-copy\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756736 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-daemon-config\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756812 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxr52\" (UniqueName: \"kubernetes.io/projected/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kube-api-access-rxr52\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756840 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756858 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-iptables-alerter-script\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756888 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-script-lib\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756915 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svzp\" (UniqueName: \"kubernetes.io/projected/1e17c916-ecd2-4ccd-bfeb-451380f47d30-kube-api-access-4svzp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756927 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-cni-binary-copy\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756941 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-etc-kubernetes\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.756933 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-daemon-config\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757048 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-iptables-alerter-script\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:01:37.757058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757054 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-dbus\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757087 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757113 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92b5617d-95cb-44c2-84b5-a73a50c2d749-crio-socket\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757137 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-netns\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757164 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757186 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baba1acd-b516-4149-a337-d56b9e74b36e-host\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757213 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9af6745a-681c-44b6-861c-310df033cfe9-ovnkube-script-lib\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757235 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e75b735-0229-429d-927f-5683bcd373d6-metrics-tls\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757260 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757273 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ffacfcf-65b1-4809-8930-3c2952a90fc8-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757273 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757317 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757346 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fbdb17a3-3ead-4c90-9922-31fbae907537-hosts-file\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757467 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdb17a3-3ead-4c90-9922-31fbae907537-tmp-dir\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757467 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e75b735-0229-429d-927f-5683bcd373d6-metrics-tls\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757563 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbdb17a3-3ead-4c90-9922-31fbae907537-tmp-dir\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:01:37.757918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.757567 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a9998949-10c8-4511-857c-dfad4001ab59-original-pull-secret\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:37.761144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.761130 4711 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:01:37.761208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.761151 4711 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:01:37.761208 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:01:37.761179 4711 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:37.761208 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:01:37.761179 4711 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-5.us-east-2.compute.internal" Nov 03 20:01:37.772687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.772668 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr474\" (UniqueName: \"kubernetes.io/projected/7e75b735-0229-429d-927f-5683bcd373d6-kube-api-access-tr474\") pod \"dns-default-c8hzq\" (UID: \"7e75b735-0229-429d-927f-5683bcd373d6\") " pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:37.777091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.777076 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bm4n\" (UniqueName: \"kubernetes.io/projected/2ffacfcf-65b1-4809-8930-3c2952a90fc8-kube-api-access-4bm4n\") pod \"prometheus-operator-9cd89b74c-zb6c5\" (UID: \"2ffacfcf-65b1-4809-8930-3c2952a90fc8\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-zb6c5" Nov 03 20:01:37.777713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.777698 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptlpz\" (UniqueName: \"kubernetes.io/projected/92b5617d-95cb-44c2-84b5-a73a50c2d749-kube-api-access-ptlpz\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.777792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.777778 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flsn\" (UniqueName: \"kubernetes.io/projected/fbdb17a3-3ead-4c90-9922-31fbae907537-kube-api-access-9flsn\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:01:37.779049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.779033 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djpvm\" (UniqueName: \"kubernetes.io/projected/91a2c232-0afd-43ed-a457-c977e4961416-kube-api-access-djpvm\") pod \"ingress-canary-7wjnx\" (UID: \"91a2c232-0afd-43ed-a457-c977e4961416\") " pod="openshift-ingress-canary/ingress-canary-7wjnx" Nov 03 20:01:37.781363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.781347 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsj6v\" (UniqueName: \"kubernetes.io/projected/d938e951-475c-47d6-8d73-f17da86400ad-kube-api-access-dsj6v\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.792736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.792681 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz86h\" (UniqueName: \"kubernetes.io/projected/baba1acd-b516-4149-a337-d56b9e74b36e-kube-api-access-dz86h\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:01:37.792849 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.792774 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8w7\" (UniqueName: \"kubernetes.io/projected/9af6745a-681c-44b6-861c-310df033cfe9-kube-api-access-7g8w7\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.792943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.792922 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gb7w\" (UniqueName: \"kubernetes.io/projected/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-kube-api-access-7gb7w\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:01:37.793002 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.792931 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7dj\" (UniqueName: \"kubernetes.io/projected/43579876-2c2b-4730-bd82-717f9b116bab-kube-api-access-hj7dj\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.793044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.793007 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svzp\" (UniqueName: \"kubernetes.io/projected/1e17c916-ecd2-4ccd-bfeb-451380f47d30-kube-api-access-4svzp\") pod \"downloads-6dd789c8b7-smhql\" (UID: \"1e17c916-ecd2-4ccd-bfeb-451380f47d30\") " pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:37.793118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.793103 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks7jx\" (UniqueName: \"kubernetes.io/projected/dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969-kube-api-access-ks7jx\") pod \"network-metrics-daemon-xf24w\" (UID: \"dd9b08ea-b7e2-4d9f-8e6a-2c363d1a8969\") " pod="openshift-multus/network-metrics-daemon-xf24w" Nov 03 20:01:37.793151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.793133 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfgx\" (UniqueName: \"kubernetes.io/projected/a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b-kube-api-access-wsfgx\") pod \"network-check-target-4xx8n\" (UID: \"a1a36f2d-4a81-45b5-92b3-cbf2d69a2b6b\") " pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:37.793245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.793231 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxr52\" (UniqueName: \"kubernetes.io/projected/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kube-api-access-rxr52\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.793274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.793248 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvms\" (UniqueName: \"kubernetes.io/projected/a3f23518-97e0-4f3f-b939-d76052dc1339-kube-api-access-sjvms\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.857773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857737 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-netd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.857773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857771 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-conf\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857790 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-system-cni-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857805 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-cnibin\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857822 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-registration-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857835 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-sys-fs\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857850 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-node-log\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857861 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-netd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857897 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-node-log\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857898 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-cnibin\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857890 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-system-cni-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857927 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-registration-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857941 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-log-socket\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857942 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-sys-fs\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857959 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-lib-modules\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857932 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-conf\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857979 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90ef7a29-08bf-4a3c-b815-a34698896df1-sys\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.857981 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-log-socket\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858013 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlvm\" (UniqueName: \"kubernetes.io/projected/90ef7a29-08bf-4a3c-b815-a34698896df1-kube-api-access-6xlvm\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858019 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90ef7a29-08bf-4a3c-b815-a34698896df1-sys\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858034 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-kubelet\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858055 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858076 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-host-slash\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858101 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858113 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-kubelet\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858127 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90ef7a29-08bf-4a3c-b815-a34698896df1-root\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858137 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2cc4b09-f2b3-4eb1-93fa-41ab773f995a-host-slash\") pod \"iptables-alerter-tffb6\" (UID: \"a2cc4b09-f2b3-4eb1-93fa-41ab773f995a\") " pod="openshift-network-operator/iptables-alerter-tffb6" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858142 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-lib-modules\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858130 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d938e951-475c-47d6-8d73-f17da86400ad-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7lv9\" (UID: \"d938e951-475c-47d6-8d73-f17da86400ad\") " pod="openshift-multus/multus-additional-cni-plugins-s7lv9" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858164 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858158 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-system-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858186 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90ef7a29-08bf-4a3c-b815-a34698896df1-root\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858199 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-k8s-cni-cncf-io\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858221 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-var-lib-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858235 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-device-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858241 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-system-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858253 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-modprobe-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858256 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-var-lib-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858245 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-k8s-cni-cncf-io\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858275 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-systemd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858290 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-bin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858293 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-device-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858303 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-multus\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858327 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-systemd-units\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858353 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-systemd\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858356 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-bin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858367 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-var-lib-cni-multus\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858358 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-etc-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858393 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-etc-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858395 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-systemd-units\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858409 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-systemd\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858422 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-modprobe-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858437 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858484 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-socket-dir-parent\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858499 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-systemd\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858512 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-conf-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858539 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-textfile\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858545 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-socket-dir-parent\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858562 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-tls\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858568 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-conf-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858585 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-cnibin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858607 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-socket-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858631 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-etc-selinux\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858650 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-cnibin\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858660 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-kubelet-config\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858706 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-socket-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858715 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-kubelet-config\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858733 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-host\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.858719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858740 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-etc-selinux\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858771 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-run\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858778 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-host\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858809 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysconfig\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858843 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-textfile\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858858 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysconfig\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858860 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-run\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858875 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-kubernetes\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858912 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-wtmp\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858938 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858936 4711 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858957 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-kubernetes\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.858966 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859001 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-netns\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859023 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-hostroot\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859033 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-wtmp\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859044 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-multus-certs\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859057 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-netns\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859072 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-hostroot\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859075 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-etc-kubernetes\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859077 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-host-run-multus-certs\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859104 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-dbus\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859116 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-multus-cni-dir\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859129 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3f23518-97e0-4f3f-b939-d76052dc1339-etc-kubernetes\") pod \"multus-56d9g\" (UID: \"a3f23518-97e0-4f3f-b939-d76052dc1339\") " pod="openshift-multus/multus-56d9g" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859133 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92b5617d-95cb-44c2-84b5-a73a50c2d749-crio-socket\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859147 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859154 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-netns\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859185 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-netns\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859203 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a9998949-10c8-4511-857c-dfad4001ab59-dbus\") pod \"global-pull-secret-syncer-bhl7x\" (UID: \"a9998949-10c8-4511-857c-dfad4001ab59\") " pod="kube-system/global-pull-secret-syncer-bhl7x" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859213 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859235 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-openvswitch\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859248 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baba1acd-b516-4149-a337-d56b9e74b36e-host\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859284 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/baba1acd-b516-4149-a337-d56b9e74b36e-host\") pod \"node-ca-5h7vn\" (UID: \"baba1acd-b516-4149-a337-d56b9e74b36e\") " pod="openshift-image-registry/node-ca-5h7vn" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859284 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/92b5617d-95cb-44c2-84b5-a73a50c2d749-crio-socket\") pod \"insights-runtime-extractor-znlzm\" (UID: \"92b5617d-95cb-44c2-84b5-a73a50c2d749\") " pod="openshift-insights/insights-runtime-extractor-znlzm" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859288 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859329 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-894nr\" (UID: \"dfcea6c8-6250-4a3e-9f1e-d4ea3c02c52f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-894nr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859356 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fbdb17a3-3ead-4c90-9922-31fbae907537-hosts-file\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859370 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-kubelet\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859401 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ef7a29-08bf-4a3c-b815-a34698896df1-metrics-client-ca\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859404 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-kubelet\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859423 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-sys\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859460 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-var-lib-kubelet\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859476 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fbdb17a3-3ead-4c90-9922-31fbae907537-hosts-file\") pod \"node-resolver-wszxg\" (UID: \"fbdb17a3-3ead-4c90-9922-31fbae907537\") " pod="openshift-dns/node-resolver-wszxg" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859484 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-sys\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859490 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-slash\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859529 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-slash\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859540 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-var-lib-kubelet\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859557 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-ovn\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859601 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-run-ovn\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859601 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859642 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859646 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-run-ovn-kubernetes\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859662 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-bin\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859719 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9af6745a-681c-44b6-861c-310df033cfe9-host-cni-bin\") pod \"ovnkube-node-txw5p\" (UID: \"9af6745a-681c-44b6-861c-310df033cfe9\") " pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.859757 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43579876-2c2b-4730-bd82-717f9b116bab-etc-sysctl-d\") pod \"tuned-qc2tr\" (UID: \"43579876-2c2b-4730-bd82-717f9b116bab\") " pod="openshift-cluster-node-tuning-operator/tuned-qc2tr" Nov 03 20:01:37.860587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.860213 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90ef7a29-08bf-4a3c-b815-a34698896df1-metrics-client-ca\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.861987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.861846 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.861987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.861916 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90ef7a29-08bf-4a3c-b815-a34698896df1-node-exporter-tls\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.879378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.879205 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlvm\" (UniqueName: \"kubernetes.io/projected/90ef7a29-08bf-4a3c-b815-a34698896df1-kube-api-access-6xlvm\") pod \"node-exporter-q2skn\" (UID: \"90ef7a29-08bf-4a3c-b815-a34698896df1\") " pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.945245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.945164 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:01:37.958099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:37.958076 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2skn" Nov 03 20:01:37.968841 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:37.968818 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ef7a29_08bf_4a3c_b815_a34698896df1.slice/crio-b93eaa6b905f5ec42fe72164f58dc6e16e1b59a59a59d2b884f3e67eeaf81f3a WatchSource:0}: Error finding container b93eaa6b905f5ec42fe72164f58dc6e16e1b59a59a59d2b884f3e67eeaf81f3a: Status 404 returned error can't find the container with id b93eaa6b905f5ec42fe72164f58dc6e16e1b59a59a59d2b884f3e67eeaf81f3a Nov 03 20:01:38.204578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.204544 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:38.314873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.314840 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:01:38.318015 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.317987 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.326265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.326241 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 20:01:38.341686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.341658 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:01:38.362859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.362827 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.362979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.362868 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.362979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.362901 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-out\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.362979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.362950 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbrkz\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-kube-api-access-rbrkz\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.363097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.363012 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.363097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.363027 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.363097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.363085 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.363173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.363107 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-web-config\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.363173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.363129 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.363173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.363154 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.363255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.363233 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.363283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.363261 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.363283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.363278 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.418332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.417479 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 20:01:38.418332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.417585 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 20:01:38.418332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.417637 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 20:01:38.418332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.417781 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-94m72\"" Nov 03 20:01:38.418332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.418096 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 20:01:38.418332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.418139 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 20:01:38.418332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.418191 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 20:01:38.418332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.418227 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 20:01:38.418332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.418298 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 20:01:38.447481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.447384 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:38.464334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464301 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464339 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464365 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-out\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464380 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbrkz\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-kube-api-access-rbrkz\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464401 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464417 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464434 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464473 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-web-config\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464807 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:01:38.464525 4711 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle podName:e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4 nodeName:}" failed. No retries permitted until 2026-11-03 20:01:38.964480315 +0000 UTC m=+2.414586353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4") : configmap references non-existent config key: ca-bundle.crt Nov 03 20:01:38.464807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464584 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464615 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464677 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464710 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.464807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.464730 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.465137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.465119 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.465618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.465595 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.468387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.468359 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.468492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.468415 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.468535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.468490 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-out\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.468948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.468908 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.469045 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.468979 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.469045 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.468997 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-web-config\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.469045 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.469030 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.469180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.469061 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.470274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.470258 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.490107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.490082 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbrkz\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-kube-api-access-rbrkz\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.501911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.501888 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:38.742541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.742460 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:38.749511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.749480 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2skn" event={"ID":"90ef7a29-08bf-4a3c-b815-a34698896df1","Type":"ContainerStarted","Data":"b93eaa6b905f5ec42fe72164f58dc6e16e1b59a59a59d2b884f3e67eeaf81f3a"} Nov 03 20:01:38.968836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.968796 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:38.969833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:38.969762 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:39.180620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.180080 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-654645b88f-f8qq9"] Nov 03 20:01:39.183392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.183373 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.205550 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.205522 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-654645b88f-f8qq9"] Nov 03 20:01:39.236681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.236651 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:01:39.252954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.252930 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 20:01:39.253131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.253113 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 20:01:39.253195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.253177 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 20:01:39.253238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.253212 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 20:01:39.253509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.253492 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5ll7s47nt50ar\"" Nov 03 20:01:39.253592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.253512 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-df4c4\"" Nov 03 20:01:39.253592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.253528 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 20:01:39.271561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.271536 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.271671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.271568 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-metrics-client-ca\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.271671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.271594 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.271671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.271612 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgj74\" (UniqueName: \"kubernetes.io/projected/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-kube-api-access-kgj74\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.271671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.271632 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-grpc-tls\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.271671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.271652 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-tls\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.272122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.271674 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.272122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.271693 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.272122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.271708 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-tmp\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.372883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.372846 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-grpc-tls\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.373055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.372900 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-tls\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.373055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.372927 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.373055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.372955 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.373055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.372980 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-tmp\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.373055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.373020 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.373055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.373045 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-metrics-client-ca\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.373387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.373078 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.373387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.373126 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgj74\" (UniqueName: \"kubernetes.io/projected/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-kube-api-access-kgj74\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.373767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.373737 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-tmp\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.374034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.373977 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-metrics-client-ca\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.377469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.377204 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.377707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.377681 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-grpc-tls\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.377795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.377707 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.377795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.377761 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-tls\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.378214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.378177 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.379835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.379814 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.390336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.390309 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgj74\" (UniqueName: \"kubernetes.io/projected/4d10424a-4e8c-4d2f-bbf9-f9b165f1685e-kube-api-access-kgj74\") pod \"thanos-querier-654645b88f-f8qq9\" (UID: \"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e\") " pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.544582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.544486 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:39.547193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.547168 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:01:39.550779 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:39.550756 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5be03f8_2dd4_4e01_a3b7_e065ffbe62b4.slice/crio-83bf7736428833069ea97553bf5260c9279289e8ca29a9279afa72db7e838a44 WatchSource:0}: Error finding container 83bf7736428833069ea97553bf5260c9279289e8ca29a9279afa72db7e838a44: Status 404 returned error can't find the container with id 83bf7736428833069ea97553bf5260c9279289e8ca29a9279afa72db7e838a44 Nov 03 20:01:39.584173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.584139 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:39.756234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.756199 4711 generic.go:356] "Generic (PLEG): container finished" podID="90ef7a29-08bf-4a3c-b815-a34698896df1" containerID="9e49694d192a92f2c162b64db00410fa895bd09b4c96a6d087df56620e206b63" exitCode=0 Nov 03 20:01:39.756700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.756292 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2skn" event={"ID":"90ef7a29-08bf-4a3c-b815-a34698896df1","Type":"ContainerDied","Data":"9e49694d192a92f2c162b64db00410fa895bd09b4c96a6d087df56620e206b63"} Nov 03 20:01:39.758125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.758103 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerStarted","Data":"83bf7736428833069ea97553bf5260c9279289e8ca29a9279afa72db7e838a44"} Nov 03 20:01:39.880692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:39.880657 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-654645b88f-f8qq9"] Nov 03 20:01:39.885703 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:39.885655 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d10424a_4e8c_4d2f_bbf9_f9b165f1685e.slice/crio-0695cafca53f055081f6f3c64d4ff6150f3d71a04b37685083a6a7492a33be7b WatchSource:0}: Error finding container 0695cafca53f055081f6f3c64d4ff6150f3d71a04b37685083a6a7492a33be7b: Status 404 returned error can't find the container with id 0695cafca53f055081f6f3c64d4ff6150f3d71a04b37685083a6a7492a33be7b Nov 03 20:01:40.184350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.184262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:40.236070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.236000 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:40.240709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.240679 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hh4rb" Nov 03 20:01:40.360724 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.360691 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:40.564795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.564712 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:40.764938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.764909 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2skn" event={"ID":"90ef7a29-08bf-4a3c-b815-a34698896df1","Type":"ContainerStarted","Data":"f235b5a1985f4eef351fc50f1f8d50fb8efcc0bce4dc304f499124fd1a6cd9a6"} Nov 03 20:01:40.765241 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.764969 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2skn" event={"ID":"90ef7a29-08bf-4a3c-b815-a34698896df1","Type":"ContainerStarted","Data":"a1c9736e371d8d2639e6de4655d8194c4736b77842f4bc263951d93605a73fdc"} Nov 03 20:01:40.767035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.766660 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" event={"ID":"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e","Type":"ContainerStarted","Data":"0695cafca53f055081f6f3c64d4ff6150f3d71a04b37685083a6a7492a33be7b"} Nov 03 20:01:40.835527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.835413 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:40.836102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.835850 4711 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:01:40.878462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:40.878385 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q2skn" podStartSLOduration=2.608169556 podStartE2EDuration="3.878365238s" totalImagesPullingTime="1.19297829s" totalInitContainerRuntime="77.217392ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:40.804272606 +0000 UTC m=+4.254378653" watchObservedRunningTime="2026-11-03 20:01:40.878365238 +0000 UTC m=+4.328471284" Nov 03 20:01:41.472822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.472614 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4"] Nov 03 20:01:41.476384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.476352 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.484727 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.484700 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 20:01:41.487484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.487439 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 20:01:41.490536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.490370 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 20:01:41.493003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.491947 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-frko2menk2k86\"" Nov 03 20:01:41.493003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.492849 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4"] Nov 03 20:01:41.493003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.492935 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6nd6x\"" Nov 03 20:01:41.493003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.492978 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 20:01:41.592623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.592582 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30310e93-8e08-41c0-9faf-b0be1bef5df7-metrics-server-audit-profiles\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.592811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.592633 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30310e93-8e08-41c0-9faf-b0be1bef5df7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.592811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.592674 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30310e93-8e08-41c0-9faf-b0be1bef5df7-client-ca-bundle\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.592811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.592708 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30310e93-8e08-41c0-9faf-b0be1bef5df7-audit-log\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.592811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.592739 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30310e93-8e08-41c0-9faf-b0be1bef5df7-secret-metrics-server-tls\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.592811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.592759 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps5s2\" (UniqueName: \"kubernetes.io/projected/30310e93-8e08-41c0-9faf-b0be1bef5df7-kube-api-access-ps5s2\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.592811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.592788 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30310e93-8e08-41c0-9faf-b0be1bef5df7-secret-metrics-server-client-certs\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.693641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.693606 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30310e93-8e08-41c0-9faf-b0be1bef5df7-audit-log\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.693788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.693662 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30310e93-8e08-41c0-9faf-b0be1bef5df7-secret-metrics-server-tls\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.693788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.693690 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps5s2\" (UniqueName: \"kubernetes.io/projected/30310e93-8e08-41c0-9faf-b0be1bef5df7-kube-api-access-ps5s2\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.693788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.693732 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30310e93-8e08-41c0-9faf-b0be1bef5df7-secret-metrics-server-client-certs\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.693788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.693776 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30310e93-8e08-41c0-9faf-b0be1bef5df7-metrics-server-audit-profiles\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.693990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.693812 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30310e93-8e08-41c0-9faf-b0be1bef5df7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.693990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.693851 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30310e93-8e08-41c0-9faf-b0be1bef5df7-client-ca-bundle\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.694126 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.694087 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/30310e93-8e08-41c0-9faf-b0be1bef5df7-audit-log\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.694929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.694873 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30310e93-8e08-41c0-9faf-b0be1bef5df7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.695165 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.695139 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/30310e93-8e08-41c0-9faf-b0be1bef5df7-metrics-server-audit-profiles\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.696650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.696621 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/30310e93-8e08-41c0-9faf-b0be1bef5df7-secret-metrics-server-tls\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.697051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.697028 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/30310e93-8e08-41c0-9faf-b0be1bef5df7-secret-metrics-server-client-certs\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.713892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.711411 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/30310e93-8e08-41c0-9faf-b0be1bef5df7-client-ca-bundle\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.713892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.711847 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps5s2\" (UniqueName: \"kubernetes.io/projected/30310e93-8e08-41c0-9faf-b0be1bef5df7-kube-api-access-ps5s2\") pod \"metrics-server-5bc95fdfb7-hjwb4\" (UID: \"30310e93-8e08-41c0-9faf-b0be1bef5df7\") " pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.752435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.752358 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c8hzq" Nov 03 20:01:41.772580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.772547 4711 generic.go:356] "Generic (PLEG): container finished" podID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerID="0de676267ef510e10b659cae7cdbe84de6c12acd65d21da4109dc444b35feb04" exitCode=0 Nov 03 20:01:41.773029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.772650 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerDied","Data":"0de676267ef510e10b659cae7cdbe84de6c12acd65d21da4109dc444b35feb04"} Nov 03 20:01:41.775435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.775403 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-72rf7"] Nov 03 20:01:41.780803 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.780781 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:41.789807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.789786 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:41.798084 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.798058 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-72rf7"] Nov 03 20:01:41.798182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.798132 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 20:01:41.798283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.798268 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hkdz2\"" Nov 03 20:01:41.840340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.840293 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:41.899998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.899742 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48141310-02b0-418f-8ee6-70cf94ca1a19-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-72rf7\" (UID: \"48141310-02b0-418f-8ee6-70cf94ca1a19\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:41.899998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.899923 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttmpw\" (UniqueName: \"kubernetes.io/projected/48141310-02b0-418f-8ee6-70cf94ca1a19-kube-api-access-ttmpw\") pod \"monitoring-plugin-77f4f76594-72rf7\" (UID: \"48141310-02b0-418f-8ee6-70cf94ca1a19\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:41.986715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:41.986662 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:42.001031 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.000993 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48141310-02b0-418f-8ee6-70cf94ca1a19-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-72rf7\" (UID: \"48141310-02b0-418f-8ee6-70cf94ca1a19\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:42.001191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.001067 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmpw\" (UniqueName: \"kubernetes.io/projected/48141310-02b0-418f-8ee6-70cf94ca1a19-kube-api-access-ttmpw\") pod \"monitoring-plugin-77f4f76594-72rf7\" (UID: \"48141310-02b0-418f-8ee6-70cf94ca1a19\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:42.007247 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.007219 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/48141310-02b0-418f-8ee6-70cf94ca1a19-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-72rf7\" (UID: \"48141310-02b0-418f-8ee6-70cf94ca1a19\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:42.013337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.013317 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmpw\" (UniqueName: \"kubernetes.io/projected/48141310-02b0-418f-8ee6-70cf94ca1a19-kube-api-access-ttmpw\") pod \"monitoring-plugin-77f4f76594-72rf7\" (UID: \"48141310-02b0-418f-8ee6-70cf94ca1a19\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:42.078960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.078932 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4"] Nov 03 20:01:42.084903 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:42.084875 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30310e93_8e08_41c0_9faf_b0be1bef5df7.slice/crio-fe818c294e7bad098106fed19734fc66915fdd5294ae497b4f1d388a02731e64 WatchSource:0}: Error finding container fe818c294e7bad098106fed19734fc66915fdd5294ae497b4f1d388a02731e64: Status 404 returned error can't find the container with id fe818c294e7bad098106fed19734fc66915fdd5294ae497b4f1d388a02731e64 Nov 03 20:01:42.101504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.101483 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:42.123066 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.122977 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:42.226203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.225523 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-687d6677d-h77qd"] Nov 03 20:01:42.229113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.229091 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.237262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.237117 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 20:01:42.237262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.237184 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 20:01:42.237262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.237218 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 20:01:42.237607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.237275 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 20:01:42.237607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.237394 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4h8rg\"" Nov 03 20:01:42.239656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.239635 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 20:01:42.242585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.242562 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 20:01:42.245657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.245601 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-687d6677d-h77qd"] Nov 03 20:01:42.304340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.304307 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-secret-telemeter-client\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.304456 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.304355 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe593514-53cf-4c0b-a669-f853d064502b-serving-certs-ca-bundle\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.304456 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.304386 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-federate-client-tls\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.304456 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.304428 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe593514-53cf-4c0b-a669-f853d064502b-metrics-client-ca\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.304620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.304483 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.304620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.304511 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe593514-53cf-4c0b-a669-f853d064502b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.304620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.304536 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-telemeter-client-tls\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.304620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.304569 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzbs\" (UniqueName: \"kubernetes.io/projected/fe593514-53cf-4c0b-a669-f853d064502b-kube-api-access-dtzbs\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.385578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.385526 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-72rf7"] Nov 03 20:01:42.388437 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:42.388406 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48141310_02b0_418f_8ee6_70cf94ca1a19.slice/crio-e8b7857d6efa4fc8f4e64bc833b3b546527f91c9ed1a80159f9b4db81e65b8f1 WatchSource:0}: Error finding container e8b7857d6efa4fc8f4e64bc833b3b546527f91c9ed1a80159f9b4db81e65b8f1: Status 404 returned error can't find the container with id e8b7857d6efa4fc8f4e64bc833b3b546527f91c9ed1a80159f9b4db81e65b8f1 Nov 03 20:01:42.405705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.405682 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-secret-telemeter-client\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.405811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.405732 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe593514-53cf-4c0b-a669-f853d064502b-serving-certs-ca-bundle\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.405811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.405762 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-federate-client-tls\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.405811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.405801 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe593514-53cf-4c0b-a669-f853d064502b-metrics-client-ca\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.405971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.405847 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.405971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.405876 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe593514-53cf-4c0b-a669-f853d064502b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.405971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.405903 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-telemeter-client-tls\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.405971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.405936 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzbs\" (UniqueName: \"kubernetes.io/projected/fe593514-53cf-4c0b-a669-f853d064502b-kube-api-access-dtzbs\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.406694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.406658 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe593514-53cf-4c0b-a669-f853d064502b-metrics-client-ca\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.408080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.408032 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe593514-53cf-4c0b-a669-f853d064502b-serving-certs-ca-bundle\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.408506 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.408485 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe593514-53cf-4c0b-a669-f853d064502b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.408912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.408865 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-secret-telemeter-client\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.409217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.409188 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.409374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.409343 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-federate-client-tls\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.411198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.411160 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fe593514-53cf-4c0b-a669-f853d064502b-telemeter-client-tls\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.424156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.424129 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzbs\" (UniqueName: \"kubernetes.io/projected/fe593514-53cf-4c0b-a669-f853d064502b-kube-api-access-dtzbs\") pod \"telemeter-client-687d6677d-h77qd\" (UID: \"fe593514-53cf-4c0b-a669-f853d064502b\") " pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.551048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.550969 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" Nov 03 20:01:42.788319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.788268 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" event={"ID":"30310e93-8e08-41c0-9faf-b0be1bef5df7","Type":"ContainerStarted","Data":"fe818c294e7bad098106fed19734fc66915fdd5294ae497b4f1d388a02731e64"} Nov 03 20:01:42.791287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.791259 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" event={"ID":"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e","Type":"ContainerStarted","Data":"d91a99c21f6d00aec7709422a9b10f810c7d7e4fb4e2922b8094bff8604e4431"} Nov 03 20:01:42.791441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.791325 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" event={"ID":"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e","Type":"ContainerStarted","Data":"0b99259a566546d15e27ad69a86880fd4a548e8d82b42b253ee56fdd545e0e88"} Nov 03 20:01:42.791521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.791464 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" event={"ID":"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e","Type":"ContainerStarted","Data":"b7c21ca051a5809c8c8e93329c5b32d4454a00051d33c4ccdd597e55fe3a8fc1"} Nov 03 20:01:42.793075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.793051 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" event={"ID":"48141310-02b0-418f-8ee6-70cf94ca1a19","Type":"ContainerStarted","Data":"e8b7857d6efa4fc8f4e64bc833b3b546527f91c9ed1a80159f9b4db81e65b8f1"} Nov 03 20:01:42.838654 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:42.838528 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-687d6677d-h77qd"] Nov 03 20:01:42.842068 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:42.842038 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe593514_53cf_4c0b_a669_f853d064502b.slice/crio-e2b4add142bcefb60060c5f7b9b06da9ecffa76c336f005eac9d57667cd8caec WatchSource:0}: Error finding container e2b4add142bcefb60060c5f7b9b06da9ecffa76c336f005eac9d57667cd8caec: Status 404 returned error can't find the container with id e2b4add142bcefb60060c5f7b9b06da9ecffa76c336f005eac9d57667cd8caec Nov 03 20:01:43.376087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.375960 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:43.490726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.487056 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:01:43.496245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.496228 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.507243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.505358 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 20:01:43.507243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.507914 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:01:43.518329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.518300 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:43.573934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.573903 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 20:01:43.574073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.573954 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 20:01:43.574073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.573989 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 20:01:43.574073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.573960 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 20:01:43.574073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.574050 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 20:01:43.574283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.574137 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 20:01:43.574283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.574154 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 20:01:43.574283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.574230 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 20:01:43.574418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.574289 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t7b9l\"" Nov 03 20:01:43.574418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.574357 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-co0bcstqp5o88\"" Nov 03 20:01:43.574418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.574393 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 20:01:43.601877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.601842 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 20:01:43.601877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.601864 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 20:01:43.611800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.611778 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 20:01:43.611942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.611923 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 20:01:43.620402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620372 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-web-config\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620419 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620466 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620490 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzxj\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-kube-api-access-nvzxj\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620517 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620543 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620577 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620599 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620653 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620680 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620700 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.620748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620731 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-config\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.621049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620769 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.621049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620797 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-config-out\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.621049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620825 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.621049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620855 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.621049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620916 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.621049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620951 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.621049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620973 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.621049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.620993 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.675836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.675804 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:43.721636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721609 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.721780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721648 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.721780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721672 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.721780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721700 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-config\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.721780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721734 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.721780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721759 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-config-out\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.721780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721779 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721805 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721849 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721884 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721908 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721932 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721968 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-web-config\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.721993 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.722018 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.722040 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzxj\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-kube-api-access-nvzxj\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.722066 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.722091 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.722122 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.722625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.722146 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.723077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.723038 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.723864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.723840 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.727557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.727535 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.729006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.728977 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.729621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.729593 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.730914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.730556 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.730914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.730603 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-config-out\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.730914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.730814 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.732576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.732544 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.736016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.735976 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.749767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.749741 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.750693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.750637 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.751169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.751134 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-config\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.751541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.751290 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.751541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.751503 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.752135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.752003 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.752135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.752045 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.752135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.752089 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-web-config\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.752593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.752354 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.753684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.753524 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzxj\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-kube-api-access-nvzxj\") pod \"prometheus-k8s-0\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:43.803718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.803681 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" event={"ID":"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e","Type":"ContainerStarted","Data":"8da432e9f92b9ce432c45c107f02d2403464d157c74e4edfe56f713a4c32d031"} Nov 03 20:01:43.804079 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.803777 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" event={"ID":"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e","Type":"ContainerStarted","Data":"3266b197c94e6ad33b8debac2851a5ddaf46563544c128bf9d22eb736ee72150"} Nov 03 20:01:43.814811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.814785 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" event={"ID":"fe593514-53cf-4c0b-a669-f853d064502b","Type":"ContainerStarted","Data":"e2b4add142bcefb60060c5f7b9b06da9ecffa76c336f005eac9d57667cd8caec"} Nov 03 20:01:43.831748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:43.831508 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:44.095554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.095322 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:44.155065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.154924 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:01:44.160958 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:01:44.160930 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d86e21_999c_46b4_a5c4_9144d3059f56.slice/crio-27ebb3d0cc9c97ff1df62c53f0684cc2d4e74419aa56e62b052344d1ff21813d WatchSource:0}: Error finding container 27ebb3d0cc9c97ff1df62c53f0684cc2d4e74419aa56e62b052344d1ff21813d: Status 404 returned error can't find the container with id 27ebb3d0cc9c97ff1df62c53f0684cc2d4e74419aa56e62b052344d1ff21813d Nov 03 20:01:44.261572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.261530 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:44.465751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.465637 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:44.645700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.645667 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:01:44.828296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.828261 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerStarted","Data":"29ec735b0fb4953737abe83af1c973d7978404160e5fa93e01a4a1b5206dff67"} Nov 03 20:01:44.828763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.828349 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerStarted","Data":"6b4814445ef5ce0cc36e1fcb5f7315ac0c4eaf95e0c2683a115c82a217be96fb"} Nov 03 20:01:44.828763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.828395 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerStarted","Data":"d4606436744d78d5221ac866e07a0058b707e46763299d47826f7d103b4d8693"} Nov 03 20:01:44.828763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.828439 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerStarted","Data":"9142b2a7ad5cb8090ac64ab0b84ca4b241536a69cc3d45ce675c0a0af9222409"} Nov 03 20:01:44.828763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.828497 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerStarted","Data":"3d1e1a58a99af6818fa4b33d52362988858140b10a1f01be31e1fe2ccc6d26a6"} Nov 03 20:01:44.828763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.828536 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerStarted","Data":"b2f1019760a76f1dedff7fbfe5a043d1847ea311ac2afa6ec373a6133845effc"} Nov 03 20:01:44.830346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.830318 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" event={"ID":"30310e93-8e08-41c0-9faf-b0be1bef5df7","Type":"ContainerStarted","Data":"d4c52cdd84b845e65bace49016a6765fe17bb9aa141656a2487f81f734e8599f"} Nov 03 20:01:44.833390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.833331 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" event={"ID":"4d10424a-4e8c-4d2f-bbf9-f9b165f1685e","Type":"ContainerStarted","Data":"b5e4ab438489982b8ebb7f6145b9e67d293bf9f8a4e5daedf5602428764989d1"} Nov 03 20:01:44.833523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.833470 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:44.834853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.834834 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" event={"ID":"48141310-02b0-418f-8ee6-70cf94ca1a19","Type":"ContainerStarted","Data":"24703605f5cf77589ab05c0e0ed6863bd64342c174896051abdfe6f39360d8c5"} Nov 03 20:01:44.835036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.835017 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:44.836274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.836257 4711 generic.go:356] "Generic (PLEG): container finished" podID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerID="516af48096749505fca2c1097d92f478fb64844fdd9852d824f1c1289a3baa45" exitCode=0 Nov 03 20:01:44.836364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.836284 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerDied","Data":"516af48096749505fca2c1097d92f478fb64844fdd9852d824f1c1289a3baa45"} Nov 03 20:01:44.836364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.836352 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerStarted","Data":"27ebb3d0cc9c97ff1df62c53f0684cc2d4e74419aa56e62b052344d1ff21813d"} Nov 03 20:01:44.839624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.839601 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" Nov 03 20:01:44.863235 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.863088 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.429541386 podStartE2EDuration="6.86307044s" totalImagesPullingTime="3.414579602s" totalInitContainerRuntime="18.949452ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:44.861154043 +0000 UTC m=+8.311260090" watchObservedRunningTime="2026-11-03 20:01:44.86307044 +0000 UTC m=+8.313176487" Nov 03 20:01:44.882261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.882188 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" podStartSLOduration=1.5652036169999999 podStartE2EDuration="3.882171539s" totalImagesPullingTime="2.316967922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:44.880974452 +0000 UTC m=+8.331080497" watchObservedRunningTime="2026-11-03 20:01:44.882171539 +0000 UTC m=+8.332277585" Nov 03 20:01:44.901900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.901845 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-72rf7" podStartSLOduration=1.571555504 podStartE2EDuration="3.90182795s" totalImagesPullingTime="2.330272446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:44.901499322 +0000 UTC m=+8.351605368" watchObservedRunningTime="2026-11-03 20:01:44.90182795 +0000 UTC m=+8.351933998" Nov 03 20:01:44.932488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:44.932405 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" podStartSLOduration=2.5804665399999998 podStartE2EDuration="5.932391464s" totalImagesPullingTime="3.351924924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:44.931172701 +0000 UTC m=+8.381278746" watchObservedRunningTime="2026-11-03 20:01:44.932391464 +0000 UTC m=+8.382497509" Nov 03 20:01:45.213385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:45.213348 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-txw5p" Nov 03 20:01:45.888906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:45.888872 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:46.022687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:46.022651 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:46.849797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:46.849745 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" event={"ID":"fe593514-53cf-4c0b-a669-f853d064502b","Type":"ContainerStarted","Data":"66d11e5427cb3778440d8ee076dd1834d2955874b5106f710b9299c782226b8d"} Nov 03 20:01:46.850715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:46.850356 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" event={"ID":"fe593514-53cf-4c0b-a669-f853d064502b","Type":"ContainerStarted","Data":"786f7da017935aa60f262e7dfc0a91a2151d716227dd0d5f4a69d0feb5ff1757"} Nov 03 20:01:46.850715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:46.850462 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" event={"ID":"fe593514-53cf-4c0b-a669-f853d064502b","Type":"ContainerStarted","Data":"29501e5b56eb94f513dc1b215316f4de640983578794ba3dcb33530e81da5720"} Nov 03 20:01:46.877987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:46.877929 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-687d6677d-h77qd" podStartSLOduration=1.458215309 podStartE2EDuration="4.877912137s" totalImagesPullingTime="3.419696828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:46.875767257 +0000 UTC m=+10.325873300" watchObservedRunningTime="2026-11-03 20:01:46.877912137 +0000 UTC m=+10.328018186" Nov 03 20:01:47.119131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:47.119024 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:47.329079 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:47.329046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:47.537237 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:01:47.537193 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:01:48.707776 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:48.707733 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:49.680912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:49.680856 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:49.813517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:49.813424 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:50.847934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:50.847905 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-654645b88f-f8qq9" Nov 03 20:01:50.977731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:50.977697 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8fpvs" Nov 03 20:01:52.885569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:52.885528 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerStarted","Data":"54febb6e0f727980d4b4a7fd0af62bfa3a5a8eadec64eac21531b9427d28ee3d"} Nov 03 20:01:52.885994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:52.885616 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerStarted","Data":"4d41a660e711b88e881903a9c637f4c79dd735bc302fe73a48e8532cef75bd19"} Nov 03 20:01:52.885994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:52.885660 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerStarted","Data":"0b090a024edde90c118b3da357294de508d7f400601afd9d81f3c44bfca697fd"} Nov 03 20:01:52.885994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:52.885712 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerStarted","Data":"ef0987162e0de8d9335bccec4dd1037659b43b22e8497a52bf79899ee7578884"} Nov 03 20:01:52.885994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:52.885750 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerStarted","Data":"bc6451a95425ec125a4b53a7ed0cbec9b3e7fec10b0dbe8bdd54b80209f0ae8c"} Nov 03 20:01:52.885994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:52.885788 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerStarted","Data":"eb930c50474eccf52d16c944c0764d48fab2abcbe95494aad23344f7b8134712"} Nov 03 20:01:52.924263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:52.924148 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.339511139 podStartE2EDuration="9.924133233s" totalImagesPullingTime="7.565918744s" totalInitContainerRuntime="18.70335ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:52.922379983 +0000 UTC m=+16.372486040" watchObservedRunningTime="2026-11-03 20:01:52.924133233 +0000 UTC m=+16.374239279" Nov 03 20:01:53.162999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:53.162962 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:53.296356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:53.296319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:53.481367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:53.481333 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:53.605593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:53.605479 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:53.832250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:53.832214 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:01:54.459097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:54.459059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:54.590295 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:54.590245 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:01:54.608101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:54.608062 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:55.280591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:55.280556 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:55.411713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:55.411679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:01:57.908779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:57.908741 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-smhql" event={"ID":"1e17c916-ecd2-4ccd-bfeb-451380f47d30","Type":"ContainerStarted","Data":"609e8b1feaaea7cce28ddfdacaf2f41ea36f5b0034ac405540b736a39aaa757d"} Nov 03 20:01:57.910558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:57.910525 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:01:57.920107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:01:57.920079 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-smhql" Nov 03 20:02:02.690689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:02.690657 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:02.819869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:02.819830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:03.783730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:03.783683 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:02:04.648380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:04.648345 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4xx8n" Nov 03 20:02:08.175813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:08.175778 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:08.299155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:08.299123 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:10.231513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:10.231477 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:10.358142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:10.358100 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:14.595709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:14.595683 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:02:14.599340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:14.599313 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5bc95fdfb7-hjwb4" Nov 03 20:02:17.532110 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:02:17.532061 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:02:18.297914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:18.297875 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:18.350280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:18.350255 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:18.847238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:18.847212 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:23.335513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:23.335436 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:23.465420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:23.465387 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:23.783892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:23.783860 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:23.906698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:23.906659 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:24.152255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:24.152173 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:24.284559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:24.284527 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:27.569818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:27.569773 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:02:27.570347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:27.570103 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="alertmanager" containerID="cri-o://b2f1019760a76f1dedff7fbfe5a043d1847ea311ac2afa6ec373a6133845effc" gracePeriod=120 Nov 03 20:02:27.570347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:27.570168 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy-web" containerID="cri-o://9142b2a7ad5cb8090ac64ab0b84ca4b241536a69cc3d45ce675c0a0af9222409" gracePeriod=120 Nov 03 20:02:27.570347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:27.570190 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy" containerID="cri-o://d4606436744d78d5221ac866e07a0058b707e46763299d47826f7d103b4d8693" gracePeriod=120 Nov 03 20:02:27.570347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:27.570208 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="config-reloader" containerID="cri-o://3d1e1a58a99af6818fa4b33d52362988858140b10a1f01be31e1fe2ccc6d26a6" gracePeriod=120 Nov 03 20:02:27.570347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:27.570224 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="prom-label-proxy" containerID="cri-o://29ec735b0fb4953737abe83af1c973d7978404160e5fa93e01a4a1b5206dff67" gracePeriod=120 Nov 03 20:02:27.570347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:27.570160 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy-metric" containerID="cri-o://6b4814445ef5ce0cc36e1fcb5f7315ac0c4eaf95e0c2683a115c82a217be96fb" gracePeriod=120 Nov 03 20:02:28.027729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027694 4711 generic.go:356] "Generic (PLEG): container finished" podID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerID="29ec735b0fb4953737abe83af1c973d7978404160e5fa93e01a4a1b5206dff67" exitCode=0 Nov 03 20:02:28.027729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027722 4711 generic.go:356] "Generic (PLEG): container finished" podID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerID="6b4814445ef5ce0cc36e1fcb5f7315ac0c4eaf95e0c2683a115c82a217be96fb" exitCode=0 Nov 03 20:02:28.027729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027732 4711 generic.go:356] "Generic (PLEG): container finished" podID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerID="d4606436744d78d5221ac866e07a0058b707e46763299d47826f7d103b4d8693" exitCode=0 Nov 03 20:02:28.027992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027741 4711 generic.go:356] "Generic (PLEG): container finished" podID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerID="3d1e1a58a99af6818fa4b33d52362988858140b10a1f01be31e1fe2ccc6d26a6" exitCode=0 Nov 03 20:02:28.027992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027751 4711 generic.go:356] "Generic (PLEG): container finished" podID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerID="b2f1019760a76f1dedff7fbfe5a043d1847ea311ac2afa6ec373a6133845effc" exitCode=0 Nov 03 20:02:28.027992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027762 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerDied","Data":"29ec735b0fb4953737abe83af1c973d7978404160e5fa93e01a4a1b5206dff67"} Nov 03 20:02:28.027992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027849 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerDied","Data":"6b4814445ef5ce0cc36e1fcb5f7315ac0c4eaf95e0c2683a115c82a217be96fb"} Nov 03 20:02:28.027992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027889 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerDied","Data":"d4606436744d78d5221ac866e07a0058b707e46763299d47826f7d103b4d8693"} Nov 03 20:02:28.027992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027926 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerDied","Data":"3d1e1a58a99af6818fa4b33d52362988858140b10a1f01be31e1fe2ccc6d26a6"} Nov 03 20:02:28.027992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:28.027969 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerDied","Data":"b2f1019760a76f1dedff7fbfe5a043d1847ea311ac2afa6ec373a6133845effc"} Nov 03 20:02:29.035419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.035380 4711 generic.go:356] "Generic (PLEG): container finished" podID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerID="9142b2a7ad5cb8090ac64ab0b84ca4b241536a69cc3d45ce675c0a0af9222409" exitCode=0 Nov 03 20:02:29.035836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.035422 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerDied","Data":"9142b2a7ad5cb8090ac64ab0b84ca4b241536a69cc3d45ce675c0a0af9222409"} Nov 03 20:02:29.336529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.336504 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:29.430717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.430683 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-kube-api-access-rbrkz\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-kube-api-access-rbrkz\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.430879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.430721 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-metrics-client-ca\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.430879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.430757 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-main-db\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.430879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.430792 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-main-tls\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.430879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.430837 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.430879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.430873 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-web-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-web-config\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.431140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.430907 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.431140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.430954 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-tls-assets\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-tls-assets\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.431140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.430984 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-cluster-tls-config\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.431140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.431014 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-out\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.431140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.431054 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.431140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.431100 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-volume\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-volume\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.431140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.431135 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\" (UID: \"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4\") " Nov 03 20:02:29.431522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.431127 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-main-db" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:29.431522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.431161 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-metrics-client-ca" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:29.431522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.431500 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:29.434011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.433741 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:29.434139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.434032 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-tls-assets" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:29.434139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.434081 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-kube-api-access-rbrkz" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "kube-api-access-rbrkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:29.434374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.434344 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-volume" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:29.434694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.434657 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-main-tls" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:29.434694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.434679 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-web" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:29.435227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.435206 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-metric" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:29.435635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.435613 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-out" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:29.437874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.437852 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-cluster-tls-config" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:29.444522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.444499 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-web-config" pod "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" (UID: "e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:29.532752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532713 4711 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532747 4711 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532759 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbrkz\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-kube-api-access-rbrkz\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532768 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-metrics-client-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532776 4711 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-main-db\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532785 4711 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-main-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532793 4711 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532822 4711 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-web-config\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532834 4711 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532848 4711 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-tls-assets\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532857 4711 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-cluster-tls-config\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532867 4711 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-config-out\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:29.532964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:29.532875 4711 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:30.042487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.042428 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4","Type":"ContainerDied","Data":"83bf7736428833069ea97553bf5260c9279289e8ca29a9279afa72db7e838a44"} Nov 03 20:02:30.042885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.042512 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.042885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.042558 4711 scope.go:122] "RemoveContainer" containerID="29ec735b0fb4953737abe83af1c973d7978404160e5fa93e01a4a1b5206dff67" Nov 03 20:02:30.050437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.050420 4711 scope.go:122] "RemoveContainer" containerID="6b4814445ef5ce0cc36e1fcb5f7315ac0c4eaf95e0c2683a115c82a217be96fb" Nov 03 20:02:30.059521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.058649 4711 scope.go:122] "RemoveContainer" containerID="d4606436744d78d5221ac866e07a0058b707e46763299d47826f7d103b4d8693" Nov 03 20:02:30.066531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.066508 4711 scope.go:122] "RemoveContainer" containerID="9142b2a7ad5cb8090ac64ab0b84ca4b241536a69cc3d45ce675c0a0af9222409" Nov 03 20:02:30.067241 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.066956 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:02:30.070865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.070786 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:02:30.073849 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.073834 4711 scope.go:122] "RemoveContainer" containerID="3d1e1a58a99af6818fa4b33d52362988858140b10a1f01be31e1fe2ccc6d26a6" Nov 03 20:02:30.080229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.080208 4711 scope.go:122] "RemoveContainer" containerID="b2f1019760a76f1dedff7fbfe5a043d1847ea311ac2afa6ec373a6133845effc" Nov 03 20:02:30.086720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.086704 4711 scope.go:122] "RemoveContainer" containerID="0de676267ef510e10b659cae7cdbe84de6c12acd65d21da4109dc444b35feb04" Nov 03 20:02:30.101056 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101034 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:02:30.101347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101334 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy-metric" Nov 03 20:02:30.101393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101349 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy-metric" Nov 03 20:02:30.101393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101364 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy-web" Nov 03 20:02:30.101393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101369 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy-web" Nov 03 20:02:30.101393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101377 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="init-config-reloader" Nov 03 20:02:30.101393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101381 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="init-config-reloader" Nov 03 20:02:30.101393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101390 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy" Nov 03 20:02:30.101393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101395 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101414 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="alertmanager" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101420 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="alertmanager" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101431 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="config-reloader" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101436 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="config-reloader" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101457 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="prom-label-proxy" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101465 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="prom-label-proxy" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101522 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101533 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="alertmanager" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101541 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="config-reloader" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101547 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="prom-label-proxy" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101553 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy-metric" Nov 03 20:02:30.101631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.101559 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" containerName="kube-rbac-proxy-web" Nov 03 20:02:30.146847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.146818 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.156572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156548 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 20:02:30.156701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156655 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 20:02:30.156701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156686 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-94m72\"" Nov 03 20:02:30.156810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156698 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 20:02:30.156810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156702 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 20:02:30.156810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156729 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 20:02:30.156810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156686 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 20:02:30.156810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156744 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 20:02:30.156810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156768 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 20:02:30.156810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.156808 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 20:02:30.159825 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.159785 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:02:30.237944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.237912 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.237949 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.237969 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs758\" (UniqueName: \"kubernetes.io/projected/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-kube-api-access-qs758\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.237997 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.238097 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-config-volume\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.238123 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-web-config\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.238186 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-config-out\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.238208 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.238227 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.238276 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.238321 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.238340 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.238428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.238366 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.338766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.338673 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.338766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.338714 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.338766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.338741 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.338776 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.338797 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs758\" (UniqueName: \"kubernetes.io/projected/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-kube-api-access-qs758\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.338824 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.339122 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-config-volume\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.339203 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-web-config\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.339275 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-config-out\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.339314 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.339360 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.339422 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.339531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.339492 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.340015 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.339878 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.340426 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.340134 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.341038 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.341016 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.341807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.341782 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.341907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.341861 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.341907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.341884 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.341907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.341890 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.342381 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.342357 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-config-volume\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.342880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.342863 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-web-config\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.343242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.343220 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.343351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.343332 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-config-out\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.343473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.343455 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.351282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.351263 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs758\" (UniqueName: \"kubernetes.io/projected/cdbf29b0-1511-48dd-b837-fbf41e7a47a4-kube-api-access-qs758\") pod \"alertmanager-main-0\" (UID: \"cdbf29b0-1511-48dd-b837-fbf41e7a47a4\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.558318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.558283 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 20:02:30.677097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.677066 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4" path="/var/lib/kubelet/pods/e5be03f8-2dd4-4e01-a3b7-e065ffbe62b4/volumes" Nov 03 20:02:30.875555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:30.875477 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 20:02:30.878538 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:02:30.878510 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbf29b0_1511_48dd_b837_fbf41e7a47a4.slice/crio-64f0849f9df64c4d884329ac4ef9fc1ec299aae4aaa8377765172d1c82a39574 WatchSource:0}: Error finding container 64f0849f9df64c4d884329ac4ef9fc1ec299aae4aaa8377765172d1c82a39574: Status 404 returned error can't find the container with id 64f0849f9df64c4d884329ac4ef9fc1ec299aae4aaa8377765172d1c82a39574 Nov 03 20:02:31.048738 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.048696 4711 generic.go:356] "Generic (PLEG): container finished" podID="cdbf29b0-1511-48dd-b837-fbf41e7a47a4" containerID="5a57b6e369c2a407cc3c24424b3410a1f66147b0156b2827c8f1a0d2c7e144ff" exitCode=0 Nov 03 20:02:31.049140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.048780 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdbf29b0-1511-48dd-b837-fbf41e7a47a4","Type":"ContainerDied","Data":"5a57b6e369c2a407cc3c24424b3410a1f66147b0156b2827c8f1a0d2c7e144ff"} Nov 03 20:02:31.049140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.048864 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdbf29b0-1511-48dd-b837-fbf41e7a47a4","Type":"ContainerStarted","Data":"64f0849f9df64c4d884329ac4ef9fc1ec299aae4aaa8377765172d1c82a39574"} Nov 03 20:02:31.823536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.823502 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:02:31.823894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.823839 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy" containerID="cri-o://4d41a660e711b88e881903a9c637f4c79dd735bc302fe73a48e8532cef75bd19" gracePeriod=600 Nov 03 20:02:31.823894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.823890 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy-web" containerID="cri-o://0b090a024edde90c118b3da357294de508d7f400601afd9d81f3c44bfca697fd" gracePeriod=600 Nov 03 20:02:31.824088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.823945 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="thanos-sidecar" containerID="cri-o://ef0987162e0de8d9335bccec4dd1037659b43b22e8497a52bf79899ee7578884" gracePeriod=600 Nov 03 20:02:31.824088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.824007 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="config-reloader" containerID="cri-o://bc6451a95425ec125a4b53a7ed0cbec9b3e7fec10b0dbe8bdd54b80209f0ae8c" gracePeriod=600 Nov 03 20:02:31.824088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.824049 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy-thanos" containerID="cri-o://54febb6e0f727980d4b4a7fd0af62bfa3a5a8eadec64eac21531b9427d28ee3d" gracePeriod=600 Nov 03 20:02:31.824231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:31.823833 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="prometheus" containerID="cri-o://eb930c50474eccf52d16c944c0764d48fab2abcbe95494aad23344f7b8134712" gracePeriod=600 Nov 03 20:02:32.059440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.059404 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdbf29b0-1511-48dd-b837-fbf41e7a47a4","Type":"ContainerStarted","Data":"f50a6d642eda5bc03b9c7bda0cbb151d2c1fade72b20d4de64654edfac900205"} Nov 03 20:02:32.059909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.059508 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdbf29b0-1511-48dd-b837-fbf41e7a47a4","Type":"ContainerStarted","Data":"33d0264de5b81d55792dd7e8e5cdd9a2505bf021d4cb67ec8a48ec6894b0cc19"} Nov 03 20:02:32.059909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.059550 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdbf29b0-1511-48dd-b837-fbf41e7a47a4","Type":"ContainerStarted","Data":"ff5d9eb7fc2f0ce7271cbf5a81b6a1bd42075639b5a5ed6466de6251f1cfa542"} Nov 03 20:02:32.059909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.059575 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdbf29b0-1511-48dd-b837-fbf41e7a47a4","Type":"ContainerStarted","Data":"769b984ef998d792a7da26123ccbec1247cdc11e9f985e2eeae80ad4204eee91"} Nov 03 20:02:32.059909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.059604 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdbf29b0-1511-48dd-b837-fbf41e7a47a4","Type":"ContainerStarted","Data":"bec0b74b3b1bdfcfcdbcca309b6b1368f14fa518f75bbfefd5be665399241067"} Nov 03 20:02:32.059909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.059631 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cdbf29b0-1511-48dd-b837-fbf41e7a47a4","Type":"ContainerStarted","Data":"4dce777d64e84896acbacc50019cf2a70bc572c04f98fbb91b84df1de04b4d01"} Nov 03 20:02:32.070184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070151 4711 generic.go:356] "Generic (PLEG): container finished" podID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerID="54febb6e0f727980d4b4a7fd0af62bfa3a5a8eadec64eac21531b9427d28ee3d" exitCode=0 Nov 03 20:02:32.070184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070179 4711 generic.go:356] "Generic (PLEG): container finished" podID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerID="4d41a660e711b88e881903a9c637f4c79dd735bc302fe73a48e8532cef75bd19" exitCode=0 Nov 03 20:02:32.070437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070191 4711 generic.go:356] "Generic (PLEG): container finished" podID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerID="0b090a024edde90c118b3da357294de508d7f400601afd9d81f3c44bfca697fd" exitCode=0 Nov 03 20:02:32.070437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070200 4711 generic.go:356] "Generic (PLEG): container finished" podID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerID="ef0987162e0de8d9335bccec4dd1037659b43b22e8497a52bf79899ee7578884" exitCode=0 Nov 03 20:02:32.070437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070210 4711 generic.go:356] "Generic (PLEG): container finished" podID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerID="bc6451a95425ec125a4b53a7ed0cbec9b3e7fec10b0dbe8bdd54b80209f0ae8c" exitCode=0 Nov 03 20:02:32.070437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070221 4711 generic.go:356] "Generic (PLEG): container finished" podID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerID="eb930c50474eccf52d16c944c0764d48fab2abcbe95494aad23344f7b8134712" exitCode=0 Nov 03 20:02:32.070437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070221 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerDied","Data":"54febb6e0f727980d4b4a7fd0af62bfa3a5a8eadec64eac21531b9427d28ee3d"} Nov 03 20:02:32.070437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070324 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerDied","Data":"4d41a660e711b88e881903a9c637f4c79dd735bc302fe73a48e8532cef75bd19"} Nov 03 20:02:32.070437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070364 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerDied","Data":"0b090a024edde90c118b3da357294de508d7f400601afd9d81f3c44bfca697fd"} Nov 03 20:02:32.070437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070402 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerDied","Data":"ef0987162e0de8d9335bccec4dd1037659b43b22e8497a52bf79899ee7578884"} Nov 03 20:02:32.070721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070440 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerDied","Data":"bc6451a95425ec125a4b53a7ed0cbec9b3e7fec10b0dbe8bdd54b80209f0ae8c"} Nov 03 20:02:32.070721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.070499 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerDied","Data":"eb930c50474eccf52d16c944c0764d48fab2abcbe95494aad23344f7b8134712"} Nov 03 20:02:32.098592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.098539 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.085382238 podStartE2EDuration="2.098520334s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.138096ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:32.097078342 +0000 UTC m=+55.547184427" watchObservedRunningTime="2026-11-03 20:02:32.098520334 +0000 UTC m=+55.548626371" Nov 03 20:02:32.130871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.130838 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:32.154721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154693 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-grpc-tls\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.154870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154736 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-metrics-client-certs\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.154870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154756 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-db\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.154870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154785 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-kubelet-serving-ca-bundle\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.154870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154807 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-kube-rbac-proxy\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.154870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154828 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-metrics-client-ca\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.154870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154862 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-web-config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-web-config\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154899 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-kube-api-access-nvzxj\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-kube-api-access-nvzxj\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154931 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-tls-assets\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-tls-assets\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154961 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-config\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.154996 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-serving-certs-ca-bundle\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155036 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-thanos-prometheus-http-client-file\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155072 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-trusted-ca-bundle\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155117 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-1\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155147 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-2\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155195 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-tls\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155200 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-kubelet-serving-ca-bundle" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:32.155686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155221 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155260 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-config-out\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-config-out\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155293 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155330 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-0\") pod \"21d86e21-999c-46b4-a5c4-9144d3059f56\" (UID: \"21d86e21-999c-46b4-a5c4-9144d3059f56\") " Nov 03 20:02:32.155944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.155819 4711 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.156345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.156128 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-db" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:32.156345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.156236 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-metrics-client-ca" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:32.160509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.160337 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-serving-certs-ca-bundle" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:32.160509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.160378 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-2" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:32.160746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.160704 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-metrics-client-certs" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:32.161500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.161000 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-1" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:32.161500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.161439 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-0" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:32.161672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.161630 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-thanos-prometheus-http-client-file" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:32.161770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.161675 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-grpc-tls" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:32.161822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.161767 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-trusted-ca-bundle" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:02:32.162134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.161923 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-tls" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:32.162234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.162203 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-tls-assets" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:32.162234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.162209 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-kube-rbac-proxy" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:32.162359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.162230 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-kube-rbac-proxy-web" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:32.162359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.162273 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-config-out" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:02:32.162359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.162330 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-thanos-sidecar-tls" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:32.162572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.162535 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-kube-api-access-nvzxj" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "kube-api-access-nvzxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:02:32.162997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.162975 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-config" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:32.172133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.172107 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-web-config" pod "21d86e21-999c-46b4-a5c4-9144d3059f56" (UID: "21d86e21-999c-46b4-a5c4-9144d3059f56"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:02:32.257073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257039 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvzxj\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-kube-api-access-nvzxj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257069 4711 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21d86e21-999c-46b4-a5c4-9144d3059f56-tls-assets\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257079 4711 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-config\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257090 4711 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257099 4711 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257108 4711 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257118 4711 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257127 4711 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257135 4711 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257144 4711 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257152 4711 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-config-out\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257160 4711 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257168 4711 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257177 4711 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-grpc-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257185 4711 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-metrics-client-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257193 4711 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21d86e21-999c-46b4-a5c4-9144d3059f56-prometheus-k8s-db\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257201 4711 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-secret-kube-rbac-proxy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257210 4711 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21d86e21-999c-46b4-a5c4-9144d3059f56-configmap-metrics-client-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:32.257289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:32.257223 4711 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21d86e21-999c-46b4-a5c4-9144d3059f56-web-config\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:02:33.077335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.077310 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.077729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.077306 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21d86e21-999c-46b4-a5c4-9144d3059f56","Type":"ContainerDied","Data":"27ebb3d0cc9c97ff1df62c53f0684cc2d4e74419aa56e62b052344d1ff21813d"} Nov 03 20:02:33.077729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.077468 4711 scope.go:122] "RemoveContainer" containerID="54febb6e0f727980d4b4a7fd0af62bfa3a5a8eadec64eac21531b9427d28ee3d" Nov 03 20:02:33.085704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.085682 4711 scope.go:122] "RemoveContainer" containerID="4d41a660e711b88e881903a9c637f4c79dd735bc302fe73a48e8532cef75bd19" Nov 03 20:02:33.092782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.092763 4711 scope.go:122] "RemoveContainer" containerID="0b090a024edde90c118b3da357294de508d7f400601afd9d81f3c44bfca697fd" Nov 03 20:02:33.099278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.099263 4711 scope.go:122] "RemoveContainer" containerID="ef0987162e0de8d9335bccec4dd1037659b43b22e8497a52bf79899ee7578884" Nov 03 20:02:33.101713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.101693 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:02:33.106696 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.106680 4711 scope.go:122] "RemoveContainer" containerID="bc6451a95425ec125a4b53a7ed0cbec9b3e7fec10b0dbe8bdd54b80209f0ae8c" Nov 03 20:02:33.107252 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.107232 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:02:33.112809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.112790 4711 scope.go:122] "RemoveContainer" containerID="eb930c50474eccf52d16c944c0764d48fab2abcbe95494aad23344f7b8134712" Nov 03 20:02:33.119618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.119602 4711 scope.go:122] "RemoveContainer" containerID="516af48096749505fca2c1097d92f478fb64844fdd9852d824f1c1289a3baa45" Nov 03 20:02:33.137744 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.137722 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:02:33.138020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138007 4711 cpu_manager.go:414] "containerMap: removing container" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="init-config-reloader" Nov 03 20:02:33.138055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138022 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="init-config-reloader" Nov 03 20:02:33.138055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138043 4711 cpu_manager.go:414] "containerMap: removing container" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="prometheus" Nov 03 20:02:33.138055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138048 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="prometheus" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138056 4711 cpu_manager.go:414] "containerMap: removing container" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy-thanos" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138061 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy-thanos" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138074 4711 cpu_manager.go:414] "containerMap: removing container" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="config-reloader" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138078 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="config-reloader" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138085 4711 cpu_manager.go:414] "containerMap: removing container" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy-web" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138089 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy-web" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138098 4711 cpu_manager.go:414] "containerMap: removing container" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138102 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138109 4711 cpu_manager.go:414] "containerMap: removing container" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="thanos-sidecar" Nov 03 20:02:33.138135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138113 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="thanos-sidecar" Nov 03 20:02:33.138388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138164 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="config-reloader" Nov 03 20:02:33.138388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138173 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy-web" Nov 03 20:02:33.138388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138179 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy" Nov 03 20:02:33.138388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138200 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="thanos-sidecar" Nov 03 20:02:33.138388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138208 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="prometheus" Nov 03 20:02:33.138388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.138214 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" containerName="kube-rbac-proxy-thanos" Nov 03 20:02:33.173248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.173223 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:02:33.173355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.173339 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.177676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.177655 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 20:02:33.177906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.177890 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 20:02:33.177964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.177893 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 20:02:33.178261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.178244 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 20:02:33.178655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.178637 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 20:02:33.178655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.178649 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 20:02:33.178772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.178659 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 20:02:33.179552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.179533 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 20:02:33.180296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.180270 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t7b9l\"" Nov 03 20:02:33.180398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.180367 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 20:02:33.181093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.181074 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 20:02:33.181315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.181301 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 20:02:33.181548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.181535 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 20:02:33.181933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.181919 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-co0bcstqp5o88\"" Nov 03 20:02:33.186808 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.186791 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 20:02:33.186928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.186914 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 20:02:33.264759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.264735 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.264843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.264762 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.264843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.264779 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.264843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.264796 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.264940 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.264856 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.264940 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.264931 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/665e6a47-3a75-4135-a005-6c2f432e7495-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.264998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.264952 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-web-config\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.264998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.264974 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.264998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.264989 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265042 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkx9w\" (UniqueName: \"kubernetes.io/projected/665e6a47-3a75-4135-a005-6c2f432e7495-kube-api-access-xkx9w\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265095 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265126 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-config\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265147 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265174 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265192 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265215 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265266 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/665e6a47-3a75-4135-a005-6c2f432e7495-config-out\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265287 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265355 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.265402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.265388 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366345 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366374 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366391 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366514 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366560 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366606 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/665e6a47-3a75-4135-a005-6c2f432e7495-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366635 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-web-config\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366749 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366782 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366815 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkx9w\" (UniqueName: \"kubernetes.io/projected/665e6a47-3a75-4135-a005-6c2f432e7495-kube-api-access-xkx9w\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366844 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.366880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366873 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-config\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366906 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366937 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.366968 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.367005 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.367033 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/665e6a47-3a75-4135-a005-6c2f432e7495-config-out\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.367059 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.367091 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.367126 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.367127 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.367313 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.367631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.367479 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.368064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.367704 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.368858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.368434 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.369218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.369102 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.369432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.369412 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.369736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.369714 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/665e6a47-3a75-4135-a005-6c2f432e7495-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.370436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.370068 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-web-config\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.370436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.370307 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.371176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.370695 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.371232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.371210 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.371828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.371767 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.371828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.371776 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.372011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.371993 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/665e6a47-3a75-4135-a005-6c2f432e7495-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.372389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.372368 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/665e6a47-3a75-4135-a005-6c2f432e7495-config-out\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.377418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.373012 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.377418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.373205 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.377418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.373227 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/665e6a47-3a75-4135-a005-6c2f432e7495-config\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.383292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.383270 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkx9w\" (UniqueName: \"kubernetes.io/projected/665e6a47-3a75-4135-a005-6c2f432e7495-kube-api-access-xkx9w\") pod \"prometheus-k8s-0\" (UID: \"665e6a47-3a75-4135-a005-6c2f432e7495\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.618367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.618287 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:33.927931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:33.927903 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 20:02:33.929162 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:02:33.929139 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665e6a47_3a75_4135_a005_6c2f432e7495.slice/crio-090b67c238e384c1e164bed5131c8a224b64f55f2be4ecc5e06fb71276905cb3 WatchSource:0}: Error finding container 090b67c238e384c1e164bed5131c8a224b64f55f2be4ecc5e06fb71276905cb3: Status 404 returned error can't find the container with id 090b67c238e384c1e164bed5131c8a224b64f55f2be4ecc5e06fb71276905cb3 Nov 03 20:02:34.085464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:34.085416 4711 generic.go:356] "Generic (PLEG): container finished" podID="665e6a47-3a75-4135-a005-6c2f432e7495" containerID="b867a6f47c2b695a7036b35573ee4e0a9442084737fff82c6dffb72e85f5a6c0" exitCode=0 Nov 03 20:02:34.085869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:34.085499 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"665e6a47-3a75-4135-a005-6c2f432e7495","Type":"ContainerDied","Data":"b867a6f47c2b695a7036b35573ee4e0a9442084737fff82c6dffb72e85f5a6c0"} Nov 03 20:02:34.085869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:34.085561 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"665e6a47-3a75-4135-a005-6c2f432e7495","Type":"ContainerStarted","Data":"090b67c238e384c1e164bed5131c8a224b64f55f2be4ecc5e06fb71276905cb3"} Nov 03 20:02:34.678062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:34.678030 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d86e21-999c-46b4-a5c4-9144d3059f56" path="/var/lib/kubelet/pods/21d86e21-999c-46b4-a5c4-9144d3059f56/volumes" Nov 03 20:02:35.094037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:35.093998 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"665e6a47-3a75-4135-a005-6c2f432e7495","Type":"ContainerStarted","Data":"c3ff2d49c220058cb75607fdde831d719578edf6815649bec8e4136810fe6949"} Nov 03 20:02:35.094473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:35.094083 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"665e6a47-3a75-4135-a005-6c2f432e7495","Type":"ContainerStarted","Data":"2883bd5a78b65e3579a5588f7d77238c511b318300bce60148b8c80d8fab62aa"} Nov 03 20:02:35.094473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:35.094125 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"665e6a47-3a75-4135-a005-6c2f432e7495","Type":"ContainerStarted","Data":"9c68345698d6094b44236dba09f6790e8edb76f3f769866515f4e574f86dbf70"} Nov 03 20:02:35.094473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:35.094162 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"665e6a47-3a75-4135-a005-6c2f432e7495","Type":"ContainerStarted","Data":"ddcf8ffa2dca4ff641b3b8bb7cda571f7da10cdf649f2da1cb90d091573c2e0f"} Nov 03 20:02:35.094473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:35.094202 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"665e6a47-3a75-4135-a005-6c2f432e7495","Type":"ContainerStarted","Data":"eeca4d071afa5b2c5da98b342ec6191bb549d7044452a71a17ac55cb6ae48ba5"} Nov 03 20:02:35.094473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:35.094235 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"665e6a47-3a75-4135-a005-6c2f432e7495","Type":"ContainerStarted","Data":"7466d0b932ffd910313e6b1b077250218275d2aad39c3d084b62ff36adfb21fa"} Nov 03 20:02:35.129933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:35.129866 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.106067171 podStartE2EDuration="2.129843497s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.776326ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:35.12875326 +0000 UTC m=+58.578859307" watchObservedRunningTime="2026-11-03 20:02:35.129843497 +0000 UTC m=+58.579949544" Nov 03 20:02:38.169930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:38.169892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:38.301043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:38.301004 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:38.619305 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:38.619277 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:02:39.082908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:39.082878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:39.226118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:39.226079 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:47.530397 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:02:47.530363 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:02:53.179115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:53.179074 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:53.310624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:53.310583 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:02:54.508777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:02:54.508737 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:04.367253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:03:04.367221 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:04.492355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:03:04.492319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:06.346123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:03:06.346074 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:03:06.361318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:03:06.361292 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:03:07.227371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:03:07.227343 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 20:03:17.529986 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:03:17.529904 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:03:34.330894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:03:34.330856 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:34.457501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:03:34.457466 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:47.530339 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:03:47.530294 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:03:47.559021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:03:47.558975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:03:54.567114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:03:54.567077 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:17.530020 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:04:17.529977 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:04:30.484470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:04:30.484421 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:04:47.529707 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:04:47.529630 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:04:54.654846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:04:54.654807 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:17.529350 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:05:17.529311 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:05:37.371893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:37.371861 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:38.829307 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:38.829270 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:47.529864 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:05:47.529829 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:05:49.507181 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:49.507138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:49.669188 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:49.669150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:50.958356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:50.958314 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:51.143357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:51.143305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:52.466308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:52.466272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:54.688770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:54.688735 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:57.677772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:57.677731 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:57.816576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:57.816544 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:57.983073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:57.982975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:58.128706 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:58.128671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:05:59.776260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.776219 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h"] Nov 03 20:05:59.778661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.778637 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:05:59.784100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.784082 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:05:59.784220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.784190 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:05:59.784324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.784304 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lvqck\"" Nov 03 20:05:59.789309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.789288 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h"] Nov 03 20:05:59.870306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.870276 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:05:59.870478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.870331 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xsml\" (UniqueName: \"kubernetes.io/projected/87e35e53-803e-4503-8d68-a63032ce6b3b-kube-api-access-5xsml\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:05:59.870569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.870483 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:05:59.970950 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.970921 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:05:59.971081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.970967 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:05:59.971081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.971009 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xsml\" (UniqueName: \"kubernetes.io/projected/87e35e53-803e-4503-8d68-a63032ce6b3b-kube-api-access-5xsml\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:05:59.971271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.971253 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:05:59.971354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.971288 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:05:59.983308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:05:59.983279 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xsml\" (UniqueName: \"kubernetes.io/projected/87e35e53-803e-4503-8d68-a63032ce6b3b-kube-api-access-5xsml\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:06:00.090743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:00.090660 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:06:00.363346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:00.363273 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h"] Nov 03 20:06:00.366241 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:06:00.366212 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e35e53_803e_4503_8d68_a63032ce6b3b.slice/crio-77a67a2c6759f59aa84f37e6de8f9c89096c1e9139a3fcd647f85fbb39d0787c WatchSource:0}: Error finding container 77a67a2c6759f59aa84f37e6de8f9c89096c1e9139a3fcd647f85fbb39d0787c: Status 404 returned error can't find the container with id 77a67a2c6759f59aa84f37e6de8f9c89096c1e9139a3fcd647f85fbb39d0787c Nov 03 20:06:00.666466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:00.666420 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:00.847217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:00.847182 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" event={"ID":"87e35e53-803e-4503-8d68-a63032ce6b3b","Type":"ContainerStarted","Data":"77a67a2c6759f59aa84f37e6de8f9c89096c1e9139a3fcd647f85fbb39d0787c"} Nov 03 20:06:05.870182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:05.870154 4711 generic.go:356] "Generic (PLEG): container finished" podID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerID="186a16b1dbc3c9dc8f416effbf52a9edb935f8ae7db34eab6749ad2610d9656a" exitCode=0 Nov 03 20:06:05.870608 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:05.870241 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" event={"ID":"87e35e53-803e-4503-8d68-a63032ce6b3b","Type":"ContainerDied","Data":"186a16b1dbc3c9dc8f416effbf52a9edb935f8ae7db34eab6749ad2610d9656a"} Nov 03 20:06:05.963230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:05.963181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:07.878273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:07.878235 4711 generic.go:356] "Generic (PLEG): container finished" podID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerID="56ce3f6ef856bc5ff51d7e0ed45bb50d8b3f0681ea207480672c2bd347f333d2" exitCode=0 Nov 03 20:06:07.878664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:07.878288 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" event={"ID":"87e35e53-803e-4503-8d68-a63032ce6b3b","Type":"ContainerDied","Data":"56ce3f6ef856bc5ff51d7e0ed45bb50d8b3f0681ea207480672c2bd347f333d2"} Nov 03 20:06:14.909465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:14.909411 4711 generic.go:356] "Generic (PLEG): container finished" podID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerID="37a2ea85a8370dd76acbed95afaf21039708936eed0a5b826a9e47b50ac8d9c1" exitCode=0 Nov 03 20:06:14.909847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:14.909499 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" event={"ID":"87e35e53-803e-4503-8d68-a63032ce6b3b","Type":"ContainerDied","Data":"37a2ea85a8370dd76acbed95afaf21039708936eed0a5b826a9e47b50ac8d9c1"} Nov 03 20:06:16.029480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.029378 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:06:16.116828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.116796 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-util\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-util\") pod \"87e35e53-803e-4503-8d68-a63032ce6b3b\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " Nov 03 20:06:16.117109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.116836 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e35e53-803e-4503-8d68-a63032ce6b3b-kube-api-access-5xsml\" (UniqueName: \"kubernetes.io/projected/87e35e53-803e-4503-8d68-a63032ce6b3b-kube-api-access-5xsml\") pod \"87e35e53-803e-4503-8d68-a63032ce6b3b\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " Nov 03 20:06:16.117109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.116868 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-bundle\") pod \"87e35e53-803e-4503-8d68-a63032ce6b3b\" (UID: \"87e35e53-803e-4503-8d68-a63032ce6b3b\") " Nov 03 20:06:16.117320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.117294 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-bundle" pod "87e35e53-803e-4503-8d68-a63032ce6b3b" (UID: "87e35e53-803e-4503-8d68-a63032ce6b3b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:16.119030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.118992 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e35e53-803e-4503-8d68-a63032ce6b3b-kube-api-access-5xsml" pod "87e35e53-803e-4503-8d68-a63032ce6b3b" (UID: "87e35e53-803e-4503-8d68-a63032ce6b3b"). InnerVolumeSpecName "kube-api-access-5xsml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:16.120428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.120407 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-util" pod "87e35e53-803e-4503-8d68-a63032ce6b3b" (UID: "87e35e53-803e-4503-8d68-a63032ce6b3b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:06:16.218437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.218366 4711 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-util\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:16.218437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.218393 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xsml\" (UniqueName: \"kubernetes.io/projected/87e35e53-803e-4503-8d68-a63032ce6b3b-kube-api-access-5xsml\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:16.218437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.218403 4711 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87e35e53-803e-4503-8d68-a63032ce6b3b-bundle\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:16.918623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.918590 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" event={"ID":"87e35e53-803e-4503-8d68-a63032ce6b3b","Type":"ContainerDied","Data":"77a67a2c6759f59aa84f37e6de8f9c89096c1e9139a3fcd647f85fbb39d0787c"} Nov 03 20:06:16.918623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.918617 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6q792h" Nov 03 20:06:16.918904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:16.918637 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77a67a2c6759f59aa84f37e6de8f9c89096c1e9139a3fcd647f85fbb39d0787c" Nov 03 20:06:17.532079 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:06:17.532048 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:06:21.441824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:21.441790 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:21.572002 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:21.571958 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:36.695648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:36.695616 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:06:36.698572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:36.698550 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:06:36.698684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:36.698551 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:06:36.701538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:36.701522 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:06:40.263379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:40.263342 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:40.394578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:40.394543 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:42.174965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.174933 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5"] Nov 03 20:06:42.175353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.175238 4711 cpu_manager.go:414] "containerMap: removing container" podUID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerName="extract" Nov 03 20:06:42.175353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.175249 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerName="extract" Nov 03 20:06:42.175353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.175263 4711 cpu_manager.go:414] "containerMap: removing container" podUID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerName="pull" Nov 03 20:06:42.175353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.175268 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerName="pull" Nov 03 20:06:42.175353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.175284 4711 cpu_manager.go:414] "containerMap: removing container" podUID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerName="util" Nov 03 20:06:42.175353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.175288 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerName="util" Nov 03 20:06:42.175353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.175340 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e35e53-803e-4503-8d68-a63032ce6b3b" containerName="extract" Nov 03 20:06:42.178508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.178491 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" Nov 03 20:06:42.187748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.187730 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:06:42.188827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.188809 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 20:06:42.188900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.188882 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-69fhq\"" Nov 03 20:06:42.192611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.192592 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5"] Nov 03 20:06:42.338522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.338482 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rg48\" (UniqueName: \"kubernetes.io/projected/92e2032e-e436-4999-b47e-e4876d1275a9-kube-api-access-2rg48\") pod \"jobset-operator-674cd59dc5-nmpj5\" (UID: \"92e2032e-e436-4999-b47e-e4876d1275a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" Nov 03 20:06:42.338692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.338623 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92e2032e-e436-4999-b47e-e4876d1275a9-tmp\") pod \"jobset-operator-674cd59dc5-nmpj5\" (UID: \"92e2032e-e436-4999-b47e-e4876d1275a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" Nov 03 20:06:42.439094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.439006 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92e2032e-e436-4999-b47e-e4876d1275a9-tmp\") pod \"jobset-operator-674cd59dc5-nmpj5\" (UID: \"92e2032e-e436-4999-b47e-e4876d1275a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" Nov 03 20:06:42.439094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.439050 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rg48\" (UniqueName: \"kubernetes.io/projected/92e2032e-e436-4999-b47e-e4876d1275a9-kube-api-access-2rg48\") pod \"jobset-operator-674cd59dc5-nmpj5\" (UID: \"92e2032e-e436-4999-b47e-e4876d1275a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" Nov 03 20:06:42.439468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.439421 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/92e2032e-e436-4999-b47e-e4876d1275a9-tmp\") pod \"jobset-operator-674cd59dc5-nmpj5\" (UID: \"92e2032e-e436-4999-b47e-e4876d1275a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" Nov 03 20:06:42.451608 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.451586 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rg48\" (UniqueName: \"kubernetes.io/projected/92e2032e-e436-4999-b47e-e4876d1275a9-kube-api-access-2rg48\") pod \"jobset-operator-674cd59dc5-nmpj5\" (UID: \"92e2032e-e436-4999-b47e-e4876d1275a9\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" Nov 03 20:06:42.497741 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.497706 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" Nov 03 20:06:42.774350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.774209 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5"] Nov 03 20:06:42.777129 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:06:42.777102 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92e2032e_e436_4999_b47e_e4876d1275a9.slice/crio-e907407638342e969733348a97852c6ae405e25e7e58dd94846d38c4da2e7e2c WatchSource:0}: Error finding container e907407638342e969733348a97852c6ae405e25e7e58dd94846d38c4da2e7e2c: Status 404 returned error can't find the container with id e907407638342e969733348a97852c6ae405e25e7e58dd94846d38c4da2e7e2c Nov 03 20:06:42.778488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:42.778470 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:06:43.019514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:43.019439 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" event={"ID":"92e2032e-e436-4999-b47e-e4876d1275a9","Type":"ContainerStarted","Data":"e907407638342e969733348a97852c6ae405e25e7e58dd94846d38c4da2e7e2c"} Nov 03 20:06:46.593277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:46.593236 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:46.733205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:46.733176 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:47.040757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:47.040723 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" event={"ID":"92e2032e-e436-4999-b47e-e4876d1275a9","Type":"ContainerStarted","Data":"87836e1f11ca5b3b5e76fbaa6bef172331b89dc03b61d483f6bbaac7648ab7d6"} Nov 03 20:06:47.063483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:47.063410 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-nmpj5" podStartSLOduration=1.140487351 podStartE2EDuration="5.063396423s" totalImagesPullingTime="3.922909072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:47.062268906 +0000 UTC m=+310.512374952" watchObservedRunningTime="2026-11-03 20:06:47.063396423 +0000 UTC m=+310.513502469" Nov 03 20:06:47.530955 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:06:47.530918 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:06:47.938614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:47.938532 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:48.059430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.059398 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll"] Nov 03 20:06:48.076467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.076424 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.092106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.092077 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0acbc67c-9146-4d15-acb4-e58197e37d99-metrics-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.092240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.092115 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0acbc67c-9146-4d15-acb4-e58197e37d99-manifests\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.092240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.092212 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0acbc67c-9146-4d15-acb4-e58197e37d99-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.092350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.092245 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xprm\" (UniqueName: \"kubernetes.io/projected/0acbc67c-9146-4d15-acb4-e58197e37d99-kube-api-access-9xprm\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.092602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.092562 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0acbc67c-9146-4d15-acb4-e58197e37d99-tmp\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.092786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.092675 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0acbc67c-9146-4d15-acb4-e58197e37d99-webhook-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.094881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.094859 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll"] Nov 03 20:06:48.119428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.119402 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:06:48.119428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.119425 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:06:48.140930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.140891 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:06:48.141094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.140955 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:06:48.141094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.141014 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-m6wx9\"" Nov 03 20:06:48.141094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.141043 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:06:48.193670 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.193588 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0acbc67c-9146-4d15-acb4-e58197e37d99-webhook-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.193670 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.193620 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0acbc67c-9146-4d15-acb4-e58197e37d99-metrics-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.193670 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.193651 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0acbc67c-9146-4d15-acb4-e58197e37d99-manifests\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.193936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.193697 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0acbc67c-9146-4d15-acb4-e58197e37d99-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.193936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.193723 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xprm\" (UniqueName: \"kubernetes.io/projected/0acbc67c-9146-4d15-acb4-e58197e37d99-kube-api-access-9xprm\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.193936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.193790 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0acbc67c-9146-4d15-acb4-e58197e37d99-tmp\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.194128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.194106 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0acbc67c-9146-4d15-acb4-e58197e37d99-manifests\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.195923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.195895 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0acbc67c-9146-4d15-acb4-e58197e37d99-tmp\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.196232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.196215 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0acbc67c-9146-4d15-acb4-e58197e37d99-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.196300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.196241 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0acbc67c-9146-4d15-acb4-e58197e37d99-metrics-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.196300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.196241 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0acbc67c-9146-4d15-acb4-e58197e37d99-webhook-cert\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.206124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.206100 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xprm\" (UniqueName: \"kubernetes.io/projected/0acbc67c-9146-4d15-acb4-e58197e37d99-kube-api-access-9xprm\") pod \"opendatahub-operator-controller-manager-8865cc5f6-5trll\" (UID: \"0acbc67c-9146-4d15-acb4-e58197e37d99\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.235977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.235942 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:48.367345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.367308 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:48.417729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.417700 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:48.508258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.508223 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:48.641914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.641883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:48.732380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:48.732353 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll"] Nov 03 20:06:48.734462 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:06:48.734425 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acbc67c_9146_4d15_acb4_e58197e37d99.slice/crio-7c2437e3e5dc9558bc31dd8ea2d7b6817193c622a3ab75b2816772fcc3dca619 WatchSource:0}: Error finding container 7c2437e3e5dc9558bc31dd8ea2d7b6817193c622a3ab75b2816772fcc3dca619: Status 404 returned error can't find the container with id 7c2437e3e5dc9558bc31dd8ea2d7b6817193c622a3ab75b2816772fcc3dca619 Nov 03 20:06:49.049683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:49.049651 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" event={"ID":"0acbc67c-9146-4d15-acb4-e58197e37d99","Type":"ContainerStarted","Data":"7c2437e3e5dc9558bc31dd8ea2d7b6817193c622a3ab75b2816772fcc3dca619"} Nov 03 20:06:50.419551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:50.419506 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:51.923853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:51.923762 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:52.065886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:52.065851 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" event={"ID":"0acbc67c-9146-4d15-acb4-e58197e37d99","Type":"ContainerStarted","Data":"ed530467c3ca761e6fa6db89742833b5a688cfb43ad6f33eaca230c21c2e044d"} Nov 03 20:06:52.120576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:52.120542 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:52.872839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:52.872806 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:53.009989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:53.009955 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:53.071348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:53.071320 4711 generic.go:356] "Generic (PLEG): container finished" podID="0acbc67c-9146-4d15-acb4-e58197e37d99" containerID="ed530467c3ca761e6fa6db89742833b5a688cfb43ad6f33eaca230c21c2e044d" exitCode=0 Nov 03 20:06:53.071511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:53.071367 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" event={"ID":"0acbc67c-9146-4d15-acb4-e58197e37d99","Type":"ContainerDied","Data":"ed530467c3ca761e6fa6db89742833b5a688cfb43ad6f33eaca230c21c2e044d"} Nov 03 20:06:53.382192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:53.382148 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:54.077141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:54.077104 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" event={"ID":"0acbc67c-9146-4d15-acb4-e58197e37d99","Type":"ContainerStarted","Data":"72449be8e7c2c6089175d52e6af2e2777f8eb3bedc99566ceb70bb80820b7686"} Nov 03 20:06:54.077599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:54.077244 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:06:54.124256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:54.124207 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" podStartSLOduration=2.4032120519999998 podStartE2EDuration="6.124192763s" totalImagesPullingTime="3.274317827s" totalInitContainerRuntime="446.662884ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:54.121579501 +0000 UTC m=+317.571685546" watchObservedRunningTime="2026-11-03 20:06:54.124192763 +0000 UTC m=+317.574298802" Nov 03 20:06:54.604121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:54.604084 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:54.748382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:54.748348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:54.934030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:54.933941 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:57.154151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.154110 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:57.424381 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.424294 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:57.474105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.474072 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg"] Nov 03 20:06:57.480730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.480708 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:06:57.487240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.486972 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 20:06:57.488197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.488177 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 20:06:57.490106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.490086 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg"] Nov 03 20:06:57.491099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.491082 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-t8f9h\"" Nov 03 20:06:57.583907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.583874 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03265159-994d-4bc4-9518-e5b0c5ad2b7e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-n8njg\" (UID: \"03265159-994d-4bc4-9518-e5b0c5ad2b7e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:06:57.584052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.583920 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncvj\" (UniqueName: \"kubernetes.io/projected/03265159-994d-4bc4-9518-e5b0c5ad2b7e-kube-api-access-vncvj\") pod \"cert-manager-webhook-5dbbbf75cc-n8njg\" (UID: \"03265159-994d-4bc4-9518-e5b0c5ad2b7e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:06:57.684598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.684526 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03265159-994d-4bc4-9518-e5b0c5ad2b7e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-n8njg\" (UID: \"03265159-994d-4bc4-9518-e5b0c5ad2b7e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:06:57.684598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.684572 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncvj\" (UniqueName: \"kubernetes.io/projected/03265159-994d-4bc4-9518-e5b0c5ad2b7e-kube-api-access-vncvj\") pod \"cert-manager-webhook-5dbbbf75cc-n8njg\" (UID: \"03265159-994d-4bc4-9518-e5b0c5ad2b7e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:06:57.698993 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.698968 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncvj\" (UniqueName: \"kubernetes.io/projected/03265159-994d-4bc4-9518-e5b0c5ad2b7e-kube-api-access-vncvj\") pod \"cert-manager-webhook-5dbbbf75cc-n8njg\" (UID: \"03265159-994d-4bc4-9518-e5b0c5ad2b7e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:06:57.699120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.699103 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03265159-994d-4bc4-9518-e5b0c5ad2b7e-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-n8njg\" (UID: \"03265159-994d-4bc4-9518-e5b0c5ad2b7e\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:06:57.793432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:57.793400 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:06:58.068716 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:58.068688 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg"] Nov 03 20:06:58.070410 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:06:58.070385 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03265159_994d_4bc4_9518_e5b0c5ad2b7e.slice/crio-9d939df3d5a4be50f717fc6b4446d20dddb9439afe3f30fc8d1aa5f4a2451cad WatchSource:0}: Error finding container 9d939df3d5a4be50f717fc6b4446d20dddb9439afe3f30fc8d1aa5f4a2451cad: Status 404 returned error can't find the container with id 9d939df3d5a4be50f717fc6b4446d20dddb9439afe3f30fc8d1aa5f4a2451cad Nov 03 20:06:58.094373 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:58.094346 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" event={"ID":"03265159-994d-4bc4-9518-e5b0c5ad2b7e","Type":"ContainerStarted","Data":"9d939df3d5a4be50f717fc6b4446d20dddb9439afe3f30fc8d1aa5f4a2451cad"} Nov 03 20:06:58.393845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:58.393767 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:59.275754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:59.275711 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:06:59.879962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:06:59.879908 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:02.114836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.114799 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" event={"ID":"03265159-994d-4bc4-9518-e5b0c5ad2b7e","Type":"ContainerStarted","Data":"b66ff085e0b2e121fd1f39b887ade6a13b483ab6ef4d555fd8be844bad27d185"} Nov 03 20:07:02.115271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.114926 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:07:02.137203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.137159 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" podStartSLOduration=1.683528204 podStartE2EDuration="5.137147201s" totalImagesPullingTime="3.453618997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:02.136307613 +0000 UTC m=+325.586413660" watchObservedRunningTime="2026-11-03 20:07:02.137147201 +0000 UTC m=+325.587253246" Nov 03 20:07:02.283758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.283725 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5"] Nov 03 20:07:02.287169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.287152 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" Nov 03 20:07:02.295303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.295275 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vljqw\"" Nov 03 20:07:02.300781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.300756 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5"] Nov 03 20:07:02.327206 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.327172 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7wf\" (UniqueName: \"kubernetes.io/projected/0b70ea0a-8122-4f1a-9a1e-26eb07074e36-kube-api-access-xz7wf\") pod \"cert-manager-cainjector-5c57bbd4fc-lf6j5\" (UID: \"0b70ea0a-8122-4f1a-9a1e-26eb07074e36\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" Nov 03 20:07:02.327365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.327228 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b70ea0a-8122-4f1a-9a1e-26eb07074e36-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lf6j5\" (UID: \"0b70ea0a-8122-4f1a-9a1e-26eb07074e36\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" Nov 03 20:07:02.428602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.428523 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b70ea0a-8122-4f1a-9a1e-26eb07074e36-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lf6j5\" (UID: \"0b70ea0a-8122-4f1a-9a1e-26eb07074e36\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" Nov 03 20:07:02.428602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.428597 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7wf\" (UniqueName: \"kubernetes.io/projected/0b70ea0a-8122-4f1a-9a1e-26eb07074e36-kube-api-access-xz7wf\") pod \"cert-manager-cainjector-5c57bbd4fc-lf6j5\" (UID: \"0b70ea0a-8122-4f1a-9a1e-26eb07074e36\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" Nov 03 20:07:02.440892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.440866 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7wf\" (UniqueName: \"kubernetes.io/projected/0b70ea0a-8122-4f1a-9a1e-26eb07074e36-kube-api-access-xz7wf\") pod \"cert-manager-cainjector-5c57bbd4fc-lf6j5\" (UID: \"0b70ea0a-8122-4f1a-9a1e-26eb07074e36\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" Nov 03 20:07:02.441000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.440867 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0b70ea0a-8122-4f1a-9a1e-26eb07074e36-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lf6j5\" (UID: \"0b70ea0a-8122-4f1a-9a1e-26eb07074e36\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" Nov 03 20:07:02.597757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.597724 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" Nov 03 20:07:02.597981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.597955 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:02.867189 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:02.867155 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5"] Nov 03 20:07:02.870792 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:07:02.870761 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b70ea0a_8122_4f1a_9a1e_26eb07074e36.slice/crio-e086809737452fb46d2cfdb3247d25284d6811a3d2bcafbfbc478c532424b225 WatchSource:0}: Error finding container e086809737452fb46d2cfdb3247d25284d6811a3d2bcafbfbc478c532424b225: Status 404 returned error can't find the container with id e086809737452fb46d2cfdb3247d25284d6811a3d2bcafbfbc478c532424b225 Nov 03 20:07:03.121047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:03.120961 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" event={"ID":"0b70ea0a-8122-4f1a-9a1e-26eb07074e36","Type":"ContainerStarted","Data":"308044bcccfe56b496278ba52894dfcff228e651eb1f7d524a01b4b2fd01b3ca"} Nov 03 20:07:03.121047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:03.121012 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" event={"ID":"0b70ea0a-8122-4f1a-9a1e-26eb07074e36","Type":"ContainerStarted","Data":"e086809737452fb46d2cfdb3247d25284d6811a3d2bcafbfbc478c532424b225"} Nov 03 20:07:03.141981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:03.141901 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lf6j5" podStartSLOduration=1.141887005 podStartE2EDuration="1.141887005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:03.14042676 +0000 UTC m=+326.590532802" watchObservedRunningTime="2026-11-03 20:07:03.141887005 +0000 UTC m=+326.591993051" Nov 03 20:07:03.599097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:03.599062 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:04.405242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:04.405207 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:04.543471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:04.543426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:05.083301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:05.083271 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8865cc5f6-5trll" Nov 03 20:07:05.424895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:05.424788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:05.588574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:05.588535 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:07.679097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:07.679060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:07.817172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:07.817134 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:07.955689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:07.955601 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:08.123493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:08.123430 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-n8njg" Nov 03 20:07:08.903335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:08.903300 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-4vrwb"] Nov 03 20:07:08.906719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:08.906701 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4vrwb" Nov 03 20:07:08.919723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:08.919689 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-z4s2g\"" Nov 03 20:07:08.920969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:08.920945 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4vrwb"] Nov 03 20:07:08.981196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:08.981161 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqct\" (UniqueName: \"kubernetes.io/projected/94e701d9-de28-464f-9427-87ea276e8077-kube-api-access-csqct\") pod \"cert-manager-597b95cb84-4vrwb\" (UID: \"94e701d9-de28-464f-9427-87ea276e8077\") " pod="cert-manager/cert-manager-597b95cb84-4vrwb" Nov 03 20:07:08.981360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:08.981287 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94e701d9-de28-464f-9427-87ea276e8077-bound-sa-token\") pod \"cert-manager-597b95cb84-4vrwb\" (UID: \"94e701d9-de28-464f-9427-87ea276e8077\") " pod="cert-manager/cert-manager-597b95cb84-4vrwb" Nov 03 20:07:09.082652 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:09.082594 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqct\" (UniqueName: \"kubernetes.io/projected/94e701d9-de28-464f-9427-87ea276e8077-kube-api-access-csqct\") pod \"cert-manager-597b95cb84-4vrwb\" (UID: \"94e701d9-de28-464f-9427-87ea276e8077\") " pod="cert-manager/cert-manager-597b95cb84-4vrwb" Nov 03 20:07:09.082842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:09.082710 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94e701d9-de28-464f-9427-87ea276e8077-bound-sa-token\") pod \"cert-manager-597b95cb84-4vrwb\" (UID: \"94e701d9-de28-464f-9427-87ea276e8077\") " pod="cert-manager/cert-manager-597b95cb84-4vrwb" Nov 03 20:07:09.095418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:09.095395 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94e701d9-de28-464f-9427-87ea276e8077-bound-sa-token\") pod \"cert-manager-597b95cb84-4vrwb\" (UID: \"94e701d9-de28-464f-9427-87ea276e8077\") " pod="cert-manager/cert-manager-597b95cb84-4vrwb" Nov 03 20:07:09.095509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:09.095461 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqct\" (UniqueName: \"kubernetes.io/projected/94e701d9-de28-464f-9427-87ea276e8077-kube-api-access-csqct\") pod \"cert-manager-597b95cb84-4vrwb\" (UID: \"94e701d9-de28-464f-9427-87ea276e8077\") " pod="cert-manager/cert-manager-597b95cb84-4vrwb" Nov 03 20:07:09.218145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:09.218034 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4vrwb" Nov 03 20:07:09.491183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:09.491111 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4vrwb"] Nov 03 20:07:09.493835 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:07:09.493801 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94e701d9_de28_464f_9427_87ea276e8077.slice/crio-f3a3ad46410c2ae788201afa0e13e24a16ad7828309c8c307528389fa9a20e71 WatchSource:0}: Error finding container f3a3ad46410c2ae788201afa0e13e24a16ad7828309c8c307528389fa9a20e71: Status 404 returned error can't find the container with id f3a3ad46410c2ae788201afa0e13e24a16ad7828309c8c307528389fa9a20e71 Nov 03 20:07:09.842461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:09.842407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:10.151315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:10.151223 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4vrwb" event={"ID":"94e701d9-de28-464f-9427-87ea276e8077","Type":"ContainerStarted","Data":"95add73a75bc3a1d10515bf9e7a70840b9e1aefcb2fe4a7e9045aa129cd2ba95"} Nov 03 20:07:10.151315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:10.151261 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4vrwb" event={"ID":"94e701d9-de28-464f-9427-87ea276e8077","Type":"ContainerStarted","Data":"f3a3ad46410c2ae788201afa0e13e24a16ad7828309c8c307528389fa9a20e71"} Nov 03 20:07:10.184177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:10.184128 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-4vrwb" podStartSLOduration=2.184114417 podStartE2EDuration="2.184114417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:10.182956492 +0000 UTC m=+333.633062537" watchObservedRunningTime="2026-11-03 20:07:10.184114417 +0000 UTC m=+333.634220463" Nov 03 20:07:11.614072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:11.614033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:11.754047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:11.754010 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:12.188046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:12.188011 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:12.322717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:12.322680 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:12.805405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:12.805370 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:13.023198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:13.023166 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:17.530215 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:07:17.530176 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:07:18.233190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:18.233158 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:18.399390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:18.399350 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:21.822634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:21.822594 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:21.948776 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:21.948744 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:22.510503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.510438 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:22.652898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.652864 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:22.777379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.777299 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-594f764c95-g9xdm"] Nov 03 20:07:22.784653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.784621 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:22.812271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.812250 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 20:07:22.820319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.820291 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:07:22.820496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.820439 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:07:22.820496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.820489 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594f764c95-g9xdm"] Nov 03 20:07:22.881243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.881214 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gkjjr\"" Nov 03 20:07:22.881613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.881321 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 20:07:22.907287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.907255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:22.909326 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.909306 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc557b1-89ba-4c1b-ae42-90b1806afcf8-tls-certs\") pod \"kube-auth-proxy-594f764c95-g9xdm\" (UID: \"9fc557b1-89ba-4c1b-ae42-90b1806afcf8\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:22.909438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.909341 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4spn8\" (UniqueName: \"kubernetes.io/projected/9fc557b1-89ba-4c1b-ae42-90b1806afcf8-kube-api-access-4spn8\") pod \"kube-auth-proxy-594f764c95-g9xdm\" (UID: \"9fc557b1-89ba-4c1b-ae42-90b1806afcf8\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:22.909557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:22.909542 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fc557b1-89ba-4c1b-ae42-90b1806afcf8-tmp\") pod \"kube-auth-proxy-594f764c95-g9xdm\" (UID: \"9fc557b1-89ba-4c1b-ae42-90b1806afcf8\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:23.010829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.010795 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fc557b1-89ba-4c1b-ae42-90b1806afcf8-tmp\") pod \"kube-auth-proxy-594f764c95-g9xdm\" (UID: \"9fc557b1-89ba-4c1b-ae42-90b1806afcf8\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:23.010994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.010835 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc557b1-89ba-4c1b-ae42-90b1806afcf8-tls-certs\") pod \"kube-auth-proxy-594f764c95-g9xdm\" (UID: \"9fc557b1-89ba-4c1b-ae42-90b1806afcf8\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:23.010994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.010870 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4spn8\" (UniqueName: \"kubernetes.io/projected/9fc557b1-89ba-4c1b-ae42-90b1806afcf8-kube-api-access-4spn8\") pod \"kube-auth-proxy-594f764c95-g9xdm\" (UID: \"9fc557b1-89ba-4c1b-ae42-90b1806afcf8\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:23.013133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.013110 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fc557b1-89ba-4c1b-ae42-90b1806afcf8-tmp\") pod \"kube-auth-proxy-594f764c95-g9xdm\" (UID: \"9fc557b1-89ba-4c1b-ae42-90b1806afcf8\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:23.013367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.013347 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fc557b1-89ba-4c1b-ae42-90b1806afcf8-tls-certs\") pod \"kube-auth-proxy-594f764c95-g9xdm\" (UID: \"9fc557b1-89ba-4c1b-ae42-90b1806afcf8\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:23.029936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.029857 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4spn8\" (UniqueName: \"kubernetes.io/projected/9fc557b1-89ba-4c1b-ae42-90b1806afcf8-kube-api-access-4spn8\") pod \"kube-auth-proxy-594f764c95-g9xdm\" (UID: \"9fc557b1-89ba-4c1b-ae42-90b1806afcf8\") " pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:23.047108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.047078 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:23.096642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.096613 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" Nov 03 20:07:23.194508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.194477 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:23.369099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:23.369026 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594f764c95-g9xdm"] Nov 03 20:07:23.371735 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:07:23.371710 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc557b1_89ba_4c1b_ae42_90b1806afcf8.slice/crio-bfc0764fd1073c2e162fd0332af654a9b84f2ec5439811ab8835cc38ee1eaaa3 WatchSource:0}: Error finding container bfc0764fd1073c2e162fd0332af654a9b84f2ec5439811ab8835cc38ee1eaaa3: Status 404 returned error can't find the container with id bfc0764fd1073c2e162fd0332af654a9b84f2ec5439811ab8835cc38ee1eaaa3 Nov 03 20:07:24.039028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.038994 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4xv5z"] Nov 03 20:07:24.043291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.042851 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:24.051227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.051202 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:07:24.052702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.052679 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4xv5z"] Nov 03 20:07:24.055015 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.054993 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 20:07:24.055113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.055087 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:07:24.060315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.060296 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nh6tb\"" Nov 03 20:07:24.139811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.139749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:24.142951 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.142851 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4xv5z"] Nov 03 20:07:24.213612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.213577 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" event={"ID":"9fc557b1-89ba-4c1b-ae42-90b1806afcf8","Type":"ContainerStarted","Data":"bfc0764fd1073c2e162fd0332af654a9b84f2ec5439811ab8835cc38ee1eaaa3"} Nov 03 20:07:24.221152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.221115 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxvq\" (UniqueName: \"kubernetes.io/projected/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-kube-api-access-lqxvq\") pod \"limitador-limitador-74dfcfb894-4xv5z\" (UID: \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:24.221439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.221330 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-config-file\") pod \"limitador-limitador-74dfcfb894-4xv5z\" (UID: \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:24.283253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.283174 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:24.322736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.322657 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-config-file\") pod \"limitador-limitador-74dfcfb894-4xv5z\" (UID: \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:24.322736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.322717 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxvq\" (UniqueName: \"kubernetes.io/projected/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-kube-api-access-lqxvq\") pod \"limitador-limitador-74dfcfb894-4xv5z\" (UID: \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:24.323387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.323363 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-config-file\") pod \"limitador-limitador-74dfcfb894-4xv5z\" (UID: \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:24.335292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.335264 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxvq\" (UniqueName: \"kubernetes.io/projected/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-kube-api-access-lqxvq\") pod \"limitador-limitador-74dfcfb894-4xv5z\" (UID: \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:24.363322 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.363270 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:24.412285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.412246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:24.548052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.548006 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:24.649124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:24.649050 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4xv5z"] Nov 03 20:07:24.652603 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:07:24.652577 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a754f4_34e2_426e_a6ae_9e2136cbbf9a.slice/crio-08517e5a308f790d50064e3d6d3f945e82204754acc61b3ee26ba5ca7b3476f6 WatchSource:0}: Error finding container 08517e5a308f790d50064e3d6d3f945e82204754acc61b3ee26ba5ca7b3476f6: Status 404 returned error can't find the container with id 08517e5a308f790d50064e3d6d3f945e82204754acc61b3ee26ba5ca7b3476f6 Nov 03 20:07:25.025197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:25.025160 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:25.191815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:25.191774 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:25.219534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:25.219491 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" event={"ID":"24a754f4-34e2-426e-a6ae-9e2136cbbf9a","Type":"ContainerStarted","Data":"08517e5a308f790d50064e3d6d3f945e82204754acc61b3ee26ba5ca7b3476f6"} Nov 03 20:07:25.380140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:25.380047 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:26.146526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.146500 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt"] Nov 03 20:07:26.154690 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.154662 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.161517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.160715 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:07:26.161517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.161404 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt"] Nov 03 20:07:26.161517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.161417 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:07:26.161517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.161503 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-wv7mk\"" Nov 03 20:07:26.164108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.164085 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 20:07:26.341833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.341798 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de0a63b3-de1f-4ead-9e75-44a17a440fc8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.342211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.341850 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de0a63b3-de1f-4ead-9e75-44a17a440fc8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.342211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.341967 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpwqq\" (UniqueName: \"kubernetes.io/projected/de0a63b3-de1f-4ead-9e75-44a17a440fc8-kube-api-access-hpwqq\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.342211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.342004 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de0a63b3-de1f-4ead-9e75-44a17a440fc8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.442855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.442780 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpwqq\" (UniqueName: \"kubernetes.io/projected/de0a63b3-de1f-4ead-9e75-44a17a440fc8-kube-api-access-hpwqq\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.442855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.442823 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de0a63b3-de1f-4ead-9e75-44a17a440fc8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.443073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.442890 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de0a63b3-de1f-4ead-9e75-44a17a440fc8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.443073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.442919 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de0a63b3-de1f-4ead-9e75-44a17a440fc8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.443279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.443243 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de0a63b3-de1f-4ead-9e75-44a17a440fc8-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.443400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.443316 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de0a63b3-de1f-4ead-9e75-44a17a440fc8-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.443821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.443803 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/de0a63b3-de1f-4ead-9e75-44a17a440fc8-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.455546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.455522 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpwqq\" (UniqueName: \"kubernetes.io/projected/de0a63b3-de1f-4ead-9e75-44a17a440fc8-kube-api-access-hpwqq\") pod \"kserve-module-controller-manager-7566cf5fc8-ncvvt\" (UID: \"de0a63b3-de1f-4ead-9e75-44a17a440fc8\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.498638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.498597 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:26.785461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:26.785348 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt"] Nov 03 20:07:26.789051 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:07:26.789022 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0a63b3_de1f_4ead_9e75_44a17a440fc8.slice/crio-5741f247ae752cbdf1994fd17f67516b6086b409468a9d2d81945964fa2e87b3 WatchSource:0}: Error finding container 5741f247ae752cbdf1994fd17f67516b6086b409468a9d2d81945964fa2e87b3: Status 404 returned error can't find the container with id 5741f247ae752cbdf1994fd17f67516b6086b409468a9d2d81945964fa2e87b3 Nov 03 20:07:27.232512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:27.232483 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" event={"ID":"de0a63b3-de1f-4ead-9e75-44a17a440fc8","Type":"ContainerStarted","Data":"5741f247ae752cbdf1994fd17f67516b6086b409468a9d2d81945964fa2e87b3"} Nov 03 20:07:27.234023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:27.233999 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" event={"ID":"24a754f4-34e2-426e-a6ae-9e2136cbbf9a","Type":"ContainerStarted","Data":"65f8925719c20eaf9713357aa2253e4cb2ab8eab726f6a122e58c7fb39063d52"} Nov 03 20:07:27.234143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:27.234124 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:27.235402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:27.235378 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" event={"ID":"9fc557b1-89ba-4c1b-ae42-90b1806afcf8","Type":"ContainerStarted","Data":"5650fb117f90b5c12a37f64450752a275cc135285e1700e4f291f0daf8a21d98"} Nov 03 20:07:27.272708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:27.272619 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" podStartSLOduration=0.762064801 podStartE2EDuration="3.272600897s" totalImagesPullingTime="2.510536096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:27.270496612 +0000 UTC m=+350.720602660" watchObservedRunningTime="2026-11-03 20:07:27.272600897 +0000 UTC m=+350.722706945" Nov 03 20:07:27.290982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:27.290929 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-594f764c95-g9xdm" podStartSLOduration=2.522163147 podStartE2EDuration="5.290916025s" totalImagesPullingTime="2.768752878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:27.289399725 +0000 UTC m=+350.739505772" watchObservedRunningTime="2026-11-03 20:07:27.290916025 +0000 UTC m=+350.741022070" Nov 03 20:07:29.246540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:29.246418 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" event={"ID":"de0a63b3-de1f-4ead-9e75-44a17a440fc8","Type":"ContainerStarted","Data":"ecde58c928945d6123a84dd8012568b613f33f7496028ead0ec7c7911719618e"} Nov 03 20:07:29.246540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:29.246505 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:29.269866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:29.269813 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" podStartSLOduration=1.186326503 podStartE2EDuration="3.269799268s" totalImagesPullingTime="2.083472765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:29.268484157 +0000 UTC m=+352.718590198" watchObservedRunningTime="2026-11-03 20:07:29.269799268 +0000 UTC m=+352.719905313" Nov 03 20:07:30.813769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:30.813733 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:30.941576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:30.941520 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:31.236514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:31.236433 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:31.367695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:31.367664 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:32.552904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:32.552861 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:32.685866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:32.685839 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:37.462156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:37.462113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:37.597017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:37.596981 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:38.241857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:38.241825 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:38.664111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:38.664080 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4xv5z"] Nov 03 20:07:38.664519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:38.664268 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" podUID="24a754f4-34e2-426e-a6ae-9e2136cbbf9a" containerName="limitador" containerID="cri-o://65f8925719c20eaf9713357aa2253e4cb2ab8eab726f6a122e58c7fb39063d52" gracePeriod=30 Nov 03 20:07:39.291404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:39.291368 4711 generic.go:356] "Generic (PLEG): container finished" podID="24a754f4-34e2-426e-a6ae-9e2136cbbf9a" containerID="65f8925719c20eaf9713357aa2253e4cb2ab8eab726f6a122e58c7fb39063d52" exitCode=0 Nov 03 20:07:39.291587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:39.291420 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" event={"ID":"24a754f4-34e2-426e-a6ae-9e2136cbbf9a","Type":"ContainerDied","Data":"65f8925719c20eaf9713357aa2253e4cb2ab8eab726f6a122e58c7fb39063d52"} Nov 03 20:07:39.601063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:39.601038 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:39.645991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:39.645964 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-kube-api-access-lqxvq\" (UniqueName: \"kubernetes.io/projected/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-kube-api-access-lqxvq\") pod \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\" (UID: \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\") " Nov 03 20:07:39.646122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:39.646061 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-config-file\" (UniqueName: \"kubernetes.io/configmap/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-config-file\") pod \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\" (UID: \"24a754f4-34e2-426e-a6ae-9e2136cbbf9a\") " Nov 03 20:07:39.646385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:39.646365 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-config-file" pod "24a754f4-34e2-426e-a6ae-9e2136cbbf9a" (UID: "24a754f4-34e2-426e-a6ae-9e2136cbbf9a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:07:39.648087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:39.648069 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-kube-api-access-lqxvq" pod "24a754f4-34e2-426e-a6ae-9e2136cbbf9a" (UID: "24a754f4-34e2-426e-a6ae-9e2136cbbf9a"). InnerVolumeSpecName "kube-api-access-lqxvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:39.746703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:39.746681 4711 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-config-file\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:39.746703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:39.746703 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqxvq\" (UniqueName: \"kubernetes.io/projected/24a754f4-34e2-426e-a6ae-9e2136cbbf9a-kube-api-access-lqxvq\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:40.296929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:40.296889 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" event={"ID":"24a754f4-34e2-426e-a6ae-9e2136cbbf9a","Type":"ContainerDied","Data":"08517e5a308f790d50064e3d6d3f945e82204754acc61b3ee26ba5ca7b3476f6"} Nov 03 20:07:40.297099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:40.296948 4711 scope.go:122] "RemoveContainer" containerID="65f8925719c20eaf9713357aa2253e4cb2ab8eab726f6a122e58c7fb39063d52" Nov 03 20:07:40.297099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:40.296906 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-4xv5z" Nov 03 20:07:40.321190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:40.321162 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4xv5z"] Nov 03 20:07:40.327928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:40.327903 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-4xv5z"] Nov 03 20:07:40.677676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:40.677588 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a754f4-34e2-426e-a6ae-9e2136cbbf9a" path="/var/lib/kubelet/pods/24a754f4-34e2-426e-a6ae-9e2136cbbf9a/volumes" Nov 03 20:07:40.679896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:40.679848 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:40.812968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:40.812929 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:41.007730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.007693 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m"] Nov 03 20:07:41.008105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.008091 4711 cpu_manager.go:414] "containerMap: removing container" podUID="24a754f4-34e2-426e-a6ae-9e2136cbbf9a" containerName="limitador" Nov 03 20:07:41.008153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.008107 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a754f4-34e2-426e-a6ae-9e2136cbbf9a" containerName="limitador" Nov 03 20:07:41.008183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.008177 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a754f4-34e2-426e-a6ae-9e2136cbbf9a" containerName="limitador" Nov 03 20:07:41.010675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.010658 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.016273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.016255 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-46fld\"" Nov 03 20:07:41.016425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.016402 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 20:07:41.016425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.016406 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 20:07:41.016425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.016418 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 20:07:41.020078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.020062 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m"] Nov 03 20:07:41.057102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.057079 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-metrics-certs\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.057203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.057127 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-manager-config\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.057203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.057177 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmcn\" (UniqueName: \"kubernetes.io/projected/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-kube-api-access-6kmcn\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.057272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.057220 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-cert\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.157687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.157654 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-cert\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.157860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.157723 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-metrics-certs\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.157860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.157738 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-manager-config\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.157860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.157838 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmcn\" (UniqueName: \"kubernetes.io/projected/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-kube-api-access-6kmcn\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.158418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.158372 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-manager-config\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.160457 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.160430 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-metrics-certs\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.160708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.160691 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-cert\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.170257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.170230 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmcn\" (UniqueName: \"kubernetes.io/projected/91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e-kube-api-access-6kmcn\") pod \"jobset-controller-manager-5d6ff584d8-t2p2m\" (UID: \"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e\") " pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.253980 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.253947 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:41.321835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.321752 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:41.379582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.379495 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:41.508705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.508669 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:41.645423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:41.645400 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m"] Nov 03 20:07:41.647098 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:07:41.647072 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cfbe7b_6c5f_4af4_bf1b_46b1a1ced40e.slice/crio-3efcb548c1fb7286d9e74debde21a5b9a11668d91ca9f31a6df86f3cb6b5f191 WatchSource:0}: Error finding container 3efcb548c1fb7286d9e74debde21a5b9a11668d91ca9f31a6df86f3cb6b5f191: Status 404 returned error can't find the container with id 3efcb548c1fb7286d9e74debde21a5b9a11668d91ca9f31a6df86f3cb6b5f191 Nov 03 20:07:42.308047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:42.308012 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" event={"ID":"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e","Type":"ContainerStarted","Data":"3efcb548c1fb7286d9e74debde21a5b9a11668d91ca9f31a6df86f3cb6b5f191"} Nov 03 20:07:44.972286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:44.972243 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:45.100565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:45.100524 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:45.749197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:45.749162 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:46.331108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:46.331029 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" event={"ID":"91cfbe7b-6c5f-4af4-bf1b-46b1a1ced40e","Type":"ContainerStarted","Data":"25d81ae14b4b6bc85ed97d2a58e755e3358a4d75a9dd885da8aeff01cc057ef4"} Nov 03 20:07:46.331497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:46.331114 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:07:46.353848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:46.353801 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" podStartSLOduration=2.186419487 podStartE2EDuration="6.353788451s" totalImagesPullingTime="4.167368964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:46.352617258 +0000 UTC m=+369.802723306" watchObservedRunningTime="2026-11-03 20:07:46.353788451 +0000 UTC m=+369.803894497" Nov 03 20:07:47.530377 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:07:47.530345 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:07:50.252386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:50.252358 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ncvvt" Nov 03 20:07:50.857927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:50.857897 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:51.004020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:51.003984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:54.803231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:54.803196 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:07:57.341858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:07:57.341824 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5d6ff584d8-t2p2m" Nov 03 20:08:05.642323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:05.642288 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:05.772839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:05.772801 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:05.900336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:05.900253 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:11.162570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:11.162531 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:11.297494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:11.297432 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:12.194643 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:12.194611 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:12.328827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:12.328791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:17.534784 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:08:17.534740 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:08:18.779154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:18.779116 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:18.938665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:18.938625 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:19.068254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:19.068168 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:19.204850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:19.204821 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:20.028199 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:20.028167 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:20.166768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:20.166736 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:20.289920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:20.289827 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:42.494236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:42.494197 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:42.622792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:42.622757 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:43.802543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.802509 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:43.871516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.871487 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd"] Nov 03 20:08:43.878726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.878705 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:43.885372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.885353 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:08:43.886923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.886908 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:08:43.886990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.886953 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:08:43.886990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.886976 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqpwn\"" Nov 03 20:08:43.887087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.886979 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:08:43.889769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.889745 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd"] Nov 03 20:08:43.927214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.927182 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b8f138-897d-46f9-aa62-2bed2a8887ce-openshift-service-ca\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:43.927372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.927225 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-maas-api-tls\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:43.927372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.927247 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5fpq\" (UniqueName: \"kubernetes.io/projected/f7b8f138-897d-46f9-aa62-2bed2a8887ce-kube-api-access-c5fpq\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:43.927372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.927334 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-metrics-certs\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:43.934481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:43.934453 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:44.028399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.028367 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b8f138-897d-46f9-aa62-2bed2a8887ce-openshift-service-ca\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:44.028564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.028419 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-maas-api-tls\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:44.028564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.028458 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5fpq\" (UniqueName: \"kubernetes.io/projected/f7b8f138-897d-46f9-aa62-2bed2a8887ce-kube-api-access-c5fpq\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:44.028564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.028495 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-metrics-certs\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:44.029064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.029043 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b8f138-897d-46f9-aa62-2bed2a8887ce-openshift-service-ca\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:44.030974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.030952 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-maas-api-tls\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:44.031064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.030996 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-metrics-certs\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:44.040900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.040878 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5fpq\" (UniqueName: \"kubernetes.io/projected/f7b8f138-897d-46f9-aa62-2bed2a8887ce-kube-api-access-c5fpq\") pod \"maas-api-7585cbf986-w4cvd\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:44.071166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.071087 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:44.192612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.192577 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:44.465144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.465122 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd"] Nov 03 20:08:44.467370 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:08:44.467344 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b8f138_897d_46f9_aa62_2bed2a8887ce.slice/crio-ce9f90397498fffa9c41bbbeadde6ddf192dc300d5c395e357e10e8b1aea923f WatchSource:0}: Error finding container ce9f90397498fffa9c41bbbeadde6ddf192dc300d5c395e357e10e8b1aea923f: Status 404 returned error can't find the container with id ce9f90397498fffa9c41bbbeadde6ddf192dc300d5c395e357e10e8b1aea923f Nov 03 20:08:44.594044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:44.594012 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" event={"ID":"f7b8f138-897d-46f9-aa62-2bed2a8887ce","Type":"ContainerStarted","Data":"ce9f90397498fffa9c41bbbeadde6ddf192dc300d5c395e357e10e8b1aea923f"} Nov 03 20:08:47.531247 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:08:47.531202 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:08:48.615272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:48.615236 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" event={"ID":"f7b8f138-897d-46f9-aa62-2bed2a8887ce","Type":"ContainerStarted","Data":"1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad"} Nov 03 20:08:48.615756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:48.615337 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:48.635311 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:48.635257 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" podStartSLOduration=1.7117518280000001 podStartE2EDuration="5.635241459s" totalImagesPullingTime="3.923489631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:48.635019134 +0000 UTC m=+432.085125191" watchObservedRunningTime="2026-11-03 20:08:48.635241459 +0000 UTC m=+432.085347506" Nov 03 20:08:54.625367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:54.625333 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:08:54.852198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:54.852162 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:54.988821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:54.988732 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:55.121055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:55.121013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:08:55.296587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:08:55.296554 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:04.846364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:04.846334 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd"] Nov 03 20:09:04.846817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:04.846533 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" podUID="f7b8f138-897d-46f9-aa62-2bed2a8887ce" containerName="maas-api" containerID="cri-o://1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad" gracePeriod=30 Nov 03 20:09:05.083606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.083579 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:09:05.219930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.219850 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-metrics-certs\") pod \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " Nov 03 20:09:05.219930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.219899 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b8f138-897d-46f9-aa62-2bed2a8887ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b8f138-897d-46f9-aa62-2bed2a8887ce-openshift-service-ca\") pod \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " Nov 03 20:09:05.220158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.219935 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-maas-api-tls\") pod \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " Nov 03 20:09:05.220158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.220038 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b8f138-897d-46f9-aa62-2bed2a8887ce-kube-api-access-c5fpq\" (UniqueName: \"kubernetes.io/projected/f7b8f138-897d-46f9-aa62-2bed2a8887ce-kube-api-access-c5fpq\") pod \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\" (UID: \"f7b8f138-897d-46f9-aa62-2bed2a8887ce\") " Nov 03 20:09:05.220362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.220328 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b8f138-897d-46f9-aa62-2bed2a8887ce-openshift-service-ca" pod "f7b8f138-897d-46f9-aa62-2bed2a8887ce" (UID: "f7b8f138-897d-46f9-aa62-2bed2a8887ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:09:05.221948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.221924 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-metrics-certs" pod "f7b8f138-897d-46f9-aa62-2bed2a8887ce" (UID: "f7b8f138-897d-46f9-aa62-2bed2a8887ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:09:05.222112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.222090 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b8f138-897d-46f9-aa62-2bed2a8887ce-kube-api-access-c5fpq" pod "f7b8f138-897d-46f9-aa62-2bed2a8887ce" (UID: "f7b8f138-897d-46f9-aa62-2bed2a8887ce"). InnerVolumeSpecName "kube-api-access-c5fpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:09:05.222176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.222142 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-maas-api-tls" pod "f7b8f138-897d-46f9-aa62-2bed2a8887ce" (UID: "f7b8f138-897d-46f9-aa62-2bed2a8887ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:09:05.321651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.321617 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5fpq\" (UniqueName: \"kubernetes.io/projected/f7b8f138-897d-46f9-aa62-2bed2a8887ce-kube-api-access-c5fpq\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:05.321651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.321646 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:05.321651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.321655 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b8f138-897d-46f9-aa62-2bed2a8887ce-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:05.321882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.321665 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7b8f138-897d-46f9-aa62-2bed2a8887ce-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:09:05.688307 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.688276 4711 generic.go:356] "Generic (PLEG): container finished" podID="f7b8f138-897d-46f9-aa62-2bed2a8887ce" containerID="1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad" exitCode=0 Nov 03 20:09:05.688500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.688316 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" Nov 03 20:09:05.688500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.688360 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" event={"ID":"f7b8f138-897d-46f9-aa62-2bed2a8887ce","Type":"ContainerDied","Data":"1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad"} Nov 03 20:09:05.688500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.688404 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd" event={"ID":"f7b8f138-897d-46f9-aa62-2bed2a8887ce","Type":"ContainerDied","Data":"ce9f90397498fffa9c41bbbeadde6ddf192dc300d5c395e357e10e8b1aea923f"} Nov 03 20:09:05.688500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.688426 4711 scope.go:122] "RemoveContainer" containerID="1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad" Nov 03 20:09:05.697303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.697282 4711 scope.go:122] "RemoveContainer" containerID="1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad" Nov 03 20:09:05.697619 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:09:05.697587 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad\": container with ID starting with 1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad not found: ID does not exist" containerID="1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad" Nov 03 20:09:05.697671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.697627 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad"} err="failed to get container status \"1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad\": rpc error: code = NotFound desc = could not find container \"1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad\": container with ID starting with 1c7517f133e5f02a0b0de9afd4c1d25382a79a902162b11b245a4b6c228f1cad not found: ID does not exist" Nov 03 20:09:05.711795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.711772 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd"] Nov 03 20:09:05.716043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:05.716023 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7585cbf986-w4cvd"] Nov 03 20:09:06.677827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:06.677791 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b8f138-897d-46f9-aa62-2bed2a8887ce" path="/var/lib/kubelet/pods/f7b8f138-897d-46f9-aa62-2bed2a8887ce/volumes" Nov 03 20:09:17.529961 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:09:17.529884 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:09:20.448313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:20.448276 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:20.576598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:20.576562 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:44.457924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:44.457884 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:09:47.530283 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:09:47.530246 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:09:54.900349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:09:54.900313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:10.654679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.654641 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:10.757292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.757255 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hjjmx"] Nov 03 20:10:10.757647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.757634 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f7b8f138-897d-46f9-aa62-2bed2a8887ce" containerName="maas-api" Nov 03 20:10:10.757696 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.757649 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b8f138-897d-46f9-aa62-2bed2a8887ce" containerName="maas-api" Nov 03 20:10:10.757735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.757701 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b8f138-897d-46f9-aa62-2bed2a8887ce" containerName="maas-api" Nov 03 20:10:10.760790 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.760772 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:10.764717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.764688 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 20:10:10.765904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.765886 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 20:10:10.767859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.767827 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-ppxrp\"" Nov 03 20:10:10.770127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.770107 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hjjmx"] Nov 03 20:10:10.788259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.788230 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:10.857875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.857846 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwss\" (UniqueName: \"kubernetes.io/projected/f2ba2b96-1e1a-445f-9068-e48bd33045e6-kube-api-access-xwwss\") pod \"ai-gateway-controller-5d8dddb676-hjjmx\" (UID: \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:10.858039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.857905 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ba2b96-1e1a-445f-9068-e48bd33045e6-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-hjjmx\" (UID: \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:10.938661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.938571 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:10.958484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.958457 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwss\" (UniqueName: \"kubernetes.io/projected/f2ba2b96-1e1a-445f-9068-e48bd33045e6-kube-api-access-xwwss\") pod \"ai-gateway-controller-5d8dddb676-hjjmx\" (UID: \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:10.958631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.958544 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ba2b96-1e1a-445f-9068-e48bd33045e6-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-hjjmx\" (UID: \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:10.960948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.960928 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ba2b96-1e1a-445f-9068-e48bd33045e6-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-hjjmx\" (UID: \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:10.971144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:10.971121 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwss\" (UniqueName: \"kubernetes.io/projected/f2ba2b96-1e1a-445f-9068-e48bd33045e6-kube-api-access-xwwss\") pod \"ai-gateway-controller-5d8dddb676-hjjmx\" (UID: \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:11.074426 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:11.074378 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:11.351768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:11.351740 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hjjmx"] Nov 03 20:10:11.353990 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:10:11.353965 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ba2b96_1e1a_445f_9068_e48bd33045e6.slice/crio-81e21a34abb62c8ce2069f097d26301035512111765ef7b937eae1ac9aa7a04d WatchSource:0}: Error finding container 81e21a34abb62c8ce2069f097d26301035512111765ef7b937eae1ac9aa7a04d: Status 404 returned error can't find the container with id 81e21a34abb62c8ce2069f097d26301035512111765ef7b937eae1ac9aa7a04d Nov 03 20:10:11.971163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:11.971129 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" event={"ID":"f2ba2b96-1e1a-445f-9068-e48bd33045e6","Type":"ContainerStarted","Data":"81e21a34abb62c8ce2069f097d26301035512111765ef7b937eae1ac9aa7a04d"} Nov 03 20:10:13.981832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:13.981797 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" event={"ID":"f2ba2b96-1e1a-445f-9068-e48bd33045e6","Type":"ContainerStarted","Data":"0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508"} Nov 03 20:10:13.982233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:13.981867 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:14.001173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.001112 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" podStartSLOduration=2.280955069 podStartE2EDuration="4.00109695s" totalImagesPullingTime="1.720141881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:14.000628416 +0000 UTC m=+517.450734477" watchObservedRunningTime="2026-11-03 20:10:14.00109695 +0000 UTC m=+517.451203007" Nov 03 20:10:14.748052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.748013 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5"] Nov 03 20:10:14.751401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.751381 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:14.759696 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.759676 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 20:10:14.762892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.762867 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5"] Nov 03 20:10:14.786856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.786827 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5pr\" (UniqueName: \"kubernetes.io/projected/5a13a60c-a9dd-4698-af2b-b6f139c83118-kube-api-access-xg5pr\") pod \"payload-pre-processing-7b98bb678c-7rjf5\" (UID: \"5a13a60c-a9dd-4698-af2b-b6f139c83118\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:14.787006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.786923 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a13a60c-a9dd-4698-af2b-b6f139c83118-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7rjf5\" (UID: \"5a13a60c-a9dd-4698-af2b-b6f139c83118\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:14.887804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.887747 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5pr\" (UniqueName: \"kubernetes.io/projected/5a13a60c-a9dd-4698-af2b-b6f139c83118-kube-api-access-xg5pr\") pod \"payload-pre-processing-7b98bb678c-7rjf5\" (UID: \"5a13a60c-a9dd-4698-af2b-b6f139c83118\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:14.887989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.887861 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a13a60c-a9dd-4698-af2b-b6f139c83118-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7rjf5\" (UID: \"5a13a60c-a9dd-4698-af2b-b6f139c83118\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:14.888631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.888599 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a13a60c-a9dd-4698-af2b-b6f139c83118-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-7rjf5\" (UID: \"5a13a60c-a9dd-4698-af2b-b6f139c83118\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:14.899865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.899833 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5pr\" (UniqueName: \"kubernetes.io/projected/5a13a60c-a9dd-4698-af2b-b6f139c83118-kube-api-access-xg5pr\") pod \"payload-pre-processing-7b98bb678c-7rjf5\" (UID: \"5a13a60c-a9dd-4698-af2b-b6f139c83118\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:14.914360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:14.914319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:15.048078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:15.048039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:15.065994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:15.065956 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:15.342959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:15.342928 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5"] Nov 03 20:10:15.345460 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:10:15.345418 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a13a60c_a9dd_4698_af2b_b6f139c83118.slice/crio-0f5f7e6971370a275d6eb2f6d0e4b642e6201624dcf43108457fcafcbc82f0e5 WatchSource:0}: Error finding container 0f5f7e6971370a275d6eb2f6d0e4b642e6201624dcf43108457fcafcbc82f0e5: Status 404 returned error can't find the container with id 0f5f7e6971370a275d6eb2f6d0e4b642e6201624dcf43108457fcafcbc82f0e5 Nov 03 20:10:15.991644 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:15.991608 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" event={"ID":"5a13a60c-a9dd-4698-af2b-b6f139c83118","Type":"ContainerStarted","Data":"0f5f7e6971370a275d6eb2f6d0e4b642e6201624dcf43108457fcafcbc82f0e5"} Nov 03 20:10:17.534057 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:10:17.534008 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:10:18.002135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:18.002040 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" event={"ID":"5a13a60c-a9dd-4698-af2b-b6f139c83118","Type":"ContainerStarted","Data":"03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5"} Nov 03 20:10:18.002278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:18.002241 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:18.021785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:18.021738 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" podStartSLOduration=1.732252946 podStartE2EDuration="4.021724802s" totalImagesPullingTime="2.289471856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:18.019836076 +0000 UTC m=+521.469942116" watchObservedRunningTime="2026-11-03 20:10:18.021724802 +0000 UTC m=+521.471830848" Nov 03 20:10:24.988975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:24.988941 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:25.246295 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.246217 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg"] Nov 03 20:10:25.249598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.249575 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:25.261949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.261924 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg"] Nov 03 20:10:25.278307 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.278281 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjts\" (UniqueName: \"kubernetes.io/projected/81a76879-bf5a-4088-a863-5e5ec0b7213b-kube-api-access-xzjts\") pod \"ai-gateway-controller-69ff9b4cb6-z65tg\" (UID: \"81a76879-bf5a-4088-a863-5e5ec0b7213b\") " pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:25.278434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.278408 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81a76879-bf5a-4088-a863-5e5ec0b7213b-metrics-certs\") pod \"ai-gateway-controller-69ff9b4cb6-z65tg\" (UID: \"81a76879-bf5a-4088-a863-5e5ec0b7213b\") " pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:25.379349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.379316 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjts\" (UniqueName: \"kubernetes.io/projected/81a76879-bf5a-4088-a863-5e5ec0b7213b-kube-api-access-xzjts\") pod \"ai-gateway-controller-69ff9b4cb6-z65tg\" (UID: \"81a76879-bf5a-4088-a863-5e5ec0b7213b\") " pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:25.379547 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.379386 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81a76879-bf5a-4088-a863-5e5ec0b7213b-metrics-certs\") pod \"ai-gateway-controller-69ff9b4cb6-z65tg\" (UID: \"81a76879-bf5a-4088-a863-5e5ec0b7213b\") " pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:25.381686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.381661 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81a76879-bf5a-4088-a863-5e5ec0b7213b-metrics-certs\") pod \"ai-gateway-controller-69ff9b4cb6-z65tg\" (UID: \"81a76879-bf5a-4088-a863-5e5ec0b7213b\") " pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:25.391391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.391369 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjts\" (UniqueName: \"kubernetes.io/projected/81a76879-bf5a-4088-a863-5e5ec0b7213b-kube-api-access-xzjts\") pod \"ai-gateway-controller-69ff9b4cb6-z65tg\" (UID: \"81a76879-bf5a-4088-a863-5e5ec0b7213b\") " pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:25.560814 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.560785 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:25.833871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:25.833848 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg"] Nov 03 20:10:25.835832 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:10:25.835802 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a76879_bf5a_4088_a863_5e5ec0b7213b.slice/crio-06428b899d2899c00cd3a9f9a87c4fe2a9d71a0688425460c886bbb24c49c465 WatchSource:0}: Error finding container 06428b899d2899c00cd3a9f9a87c4fe2a9d71a0688425460c886bbb24c49c465: Status 404 returned error can't find the container with id 06428b899d2899c00cd3a9f9a87c4fe2a9d71a0688425460c886bbb24c49c465 Nov 03 20:10:26.045693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:26.045657 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" event={"ID":"81a76879-bf5a-4088-a863-5e5ec0b7213b","Type":"ContainerStarted","Data":"06428b899d2899c00cd3a9f9a87c4fe2a9d71a0688425460c886bbb24c49c465"} Nov 03 20:10:27.051677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:27.051642 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" event={"ID":"81a76879-bf5a-4088-a863-5e5ec0b7213b","Type":"ContainerStarted","Data":"60a563fbd1078ea37bb23c25f0785794b09e14e327ef81bd6b00ea56034333a4"} Nov 03 20:10:27.052091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:27.051776 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:27.070777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:27.070612 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" podStartSLOduration=1.655367405 podStartE2EDuration="2.070597349s" totalImagesPullingTime="415.229944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:27.070410385 +0000 UTC m=+530.520516426" watchObservedRunningTime="2026-11-03 20:10:27.070597349 +0000 UTC m=+530.520703396" Nov 03 20:10:29.012400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:29.012372 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 20:10:38.061054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.061022 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-69ff9b4cb6-z65tg" Nov 03 20:10:38.107525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.107493 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hjjmx"] Nov 03 20:10:38.107748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.107723 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" podUID="f2ba2b96-1e1a-445f-9068-e48bd33045e6" containerName="manager" containerID="cri-o://0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508" gracePeriod=30 Nov 03 20:10:38.354746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.354721 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:38.492081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.492050 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ba2b96-1e1a-445f-9068-e48bd33045e6-kube-api-access-xwwss\" (UniqueName: \"kubernetes.io/projected/f2ba2b96-1e1a-445f-9068-e48bd33045e6-kube-api-access-xwwss\") pod \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\" (UID: \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\") " Nov 03 20:10:38.492268 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.492149 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ba2b96-1e1a-445f-9068-e48bd33045e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ba2b96-1e1a-445f-9068-e48bd33045e6-metrics-certs\") pod \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\" (UID: \"f2ba2b96-1e1a-445f-9068-e48bd33045e6\") " Nov 03 20:10:38.494221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.494188 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ba2b96-1e1a-445f-9068-e48bd33045e6-metrics-certs" pod "f2ba2b96-1e1a-445f-9068-e48bd33045e6" (UID: "f2ba2b96-1e1a-445f-9068-e48bd33045e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:10:38.494221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.494203 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ba2b96-1e1a-445f-9068-e48bd33045e6-kube-api-access-xwwss" pod "f2ba2b96-1e1a-445f-9068-e48bd33045e6" (UID: "f2ba2b96-1e1a-445f-9068-e48bd33045e6"). InnerVolumeSpecName "kube-api-access-xwwss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:10:38.593862 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.593793 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2ba2b96-1e1a-445f-9068-e48bd33045e6-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:38.593862 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:38.593823 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwwss\" (UniqueName: \"kubernetes.io/projected/f2ba2b96-1e1a-445f-9068-e48bd33045e6-kube-api-access-xwwss\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:10:39.119211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.119176 4711 generic.go:356] "Generic (PLEG): container finished" podID="f2ba2b96-1e1a-445f-9068-e48bd33045e6" containerID="0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508" exitCode=0 Nov 03 20:10:39.119635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.119224 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" event={"ID":"f2ba2b96-1e1a-445f-9068-e48bd33045e6","Type":"ContainerDied","Data":"0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508"} Nov 03 20:10:39.119635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.119237 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" Nov 03 20:10:39.119635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.119287 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-hjjmx" event={"ID":"f2ba2b96-1e1a-445f-9068-e48bd33045e6","Type":"ContainerDied","Data":"81e21a34abb62c8ce2069f097d26301035512111765ef7b937eae1ac9aa7a04d"} Nov 03 20:10:39.119635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.119308 4711 scope.go:122] "RemoveContainer" containerID="0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508" Nov 03 20:10:39.127632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.127612 4711 scope.go:122] "RemoveContainer" containerID="0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508" Nov 03 20:10:39.127905 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:10:39.127875 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508\": container with ID starting with 0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508 not found: ID does not exist" containerID="0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508" Nov 03 20:10:39.127960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.127910 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508"} err="failed to get container status \"0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508\": rpc error: code = NotFound desc = could not find container \"0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508\": container with ID starting with 0628a246892e809724dfa1ac2d8bda12d40f0f3fd32cd0732702222e168d7508 not found: ID does not exist" Nov 03 20:10:39.137874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.137853 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hjjmx"] Nov 03 20:10:39.141831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.141804 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-hjjmx"] Nov 03 20:10:39.836756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:39.836723 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:40.677605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:40.677571 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ba2b96-1e1a-445f-9068-e48bd33045e6" path="/var/lib/kubelet/pods/f2ba2b96-1e1a-445f-9068-e48bd33045e6/volumes" Nov 03 20:10:41.699350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:41.699317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:41.840327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:41.840297 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:47.530789 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:10:47.530713 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:10:47.536356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:47.536326 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:49.459051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.459019 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc"] Nov 03 20:10:49.459545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.459383 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f2ba2b96-1e1a-445f-9068-e48bd33045e6" containerName="manager" Nov 03 20:10:49.459545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.459393 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ba2b96-1e1a-445f-9068-e48bd33045e6" containerName="manager" Nov 03 20:10:49.459545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.459478 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ba2b96-1e1a-445f-9068-e48bd33045e6" containerName="manager" Nov 03 20:10:49.462360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.462340 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.467996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.467978 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:10:49.470728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.470584 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:10:49.472663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.472640 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vmkc9\"" Nov 03 20:10:49.473725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.473692 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 20:10:49.475890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.475867 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc"] Nov 03 20:10:49.545588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.545555 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:49.580902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.580872 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwsk\" (UniqueName: \"kubernetes.io/projected/8de1d23d-be7a-4eaa-a606-bee092c35e74-kube-api-access-hkwsk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.581026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.580959 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.581026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.581000 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de1d23d-be7a-4eaa-a606-bee092c35e74-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.581026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.581023 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.581145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.581104 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.581145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.581136 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.681581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.681550 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.681734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.681598 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de1d23d-be7a-4eaa-a606-bee092c35e74-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.681734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.681616 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.681734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.681668 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.681734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.681706 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.681933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.681738 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwsk\" (UniqueName: \"kubernetes.io/projected/8de1d23d-be7a-4eaa-a606-bee092c35e74-kube-api-access-hkwsk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.682135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.682112 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.682210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.682132 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.682210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.682156 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.684085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.684062 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de1d23d-be7a-4eaa-a606-bee092c35e74-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.684191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.684166 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de1d23d-be7a-4eaa-a606-bee092c35e74-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.693860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.693840 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwsk\" (UniqueName: \"kubernetes.io/projected/8de1d23d-be7a-4eaa-a606-bee092c35e74-kube-api-access-hkwsk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8d2rc\" (UID: \"8de1d23d-be7a-4eaa-a606-bee092c35e74\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:49.776859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:49.776837 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:50.051966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:50.051875 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc"] Nov 03 20:10:50.054189 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:10:50.054159 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de1d23d_be7a_4eaa_a606_bee092c35e74.slice/crio-0d9a0ab7cb58806c6e2b4abd10658bd613bc1b035546962e6d96edd4e701fc8c WatchSource:0}: Error finding container 0d9a0ab7cb58806c6e2b4abd10658bd613bc1b035546962e6d96edd4e701fc8c: Status 404 returned error can't find the container with id 0d9a0ab7cb58806c6e2b4abd10658bd613bc1b035546962e6d96edd4e701fc8c Nov 03 20:10:50.170675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:50.170641 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" event={"ID":"8de1d23d-be7a-4eaa-a606-bee092c35e74","Type":"ContainerStarted","Data":"0d9a0ab7cb58806c6e2b4abd10658bd613bc1b035546962e6d96edd4e701fc8c"} Nov 03 20:10:51.300492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.300468 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb"] Nov 03 20:10:51.305360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.305336 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.315334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.313214 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 20:10:51.315334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.315263 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb"] Nov 03 20:10:51.388223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.388181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:51.398275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.398247 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9gl\" (UniqueName: \"kubernetes.io/projected/669af6be-7305-4087-aafd-16fd9d48a4be-kube-api-access-6m9gl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.398399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.398296 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.398399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.398314 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669af6be-7305-4087-aafd-16fd9d48a4be-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.398489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.398463 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.398537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.398513 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.398569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.398560 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.499218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.499140 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.499218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.499177 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669af6be-7305-4087-aafd-16fd9d48a4be-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.499218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.499209 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.499433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.499247 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.499433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.499282 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.499433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.499322 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9gl\" (UniqueName: \"kubernetes.io/projected/669af6be-7305-4087-aafd-16fd9d48a4be-kube-api-access-6m9gl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.499645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.499621 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.499707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.499679 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.499762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.499719 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.501584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.501563 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/669af6be-7305-4087-aafd-16fd9d48a4be-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.501861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.501841 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/669af6be-7305-4087-aafd-16fd9d48a4be-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.511734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.511712 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9gl\" (UniqueName: \"kubernetes.io/projected/669af6be-7305-4087-aafd-16fd9d48a4be-kube-api-access-6m9gl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-lr7kb\" (UID: \"669af6be-7305-4087-aafd-16fd9d48a4be\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.621287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.621258 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:51.896030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:51.896006 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb"] Nov 03 20:10:51.898425 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:10:51.898392 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669af6be_7305_4087_aafd_16fd9d48a4be.slice/crio-8b4168abb253787d8b1cefdbd5ceca1d22984b9bae504f054c539dab16406715 WatchSource:0}: Error finding container 8b4168abb253787d8b1cefdbd5ceca1d22984b9bae504f054c539dab16406715: Status 404 returned error can't find the container with id 8b4168abb253787d8b1cefdbd5ceca1d22984b9bae504f054c539dab16406715 Nov 03 20:10:52.182297 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:52.182201 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" event={"ID":"8de1d23d-be7a-4eaa-a606-bee092c35e74","Type":"ContainerStarted","Data":"f6dac968adab0b561320798b39e70ceb7a37c38fff82ed22405f423ac24a9080"} Nov 03 20:10:52.183285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:52.183260 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" event={"ID":"669af6be-7305-4087-aafd-16fd9d48a4be","Type":"ContainerStarted","Data":"8b4168abb253787d8b1cefdbd5ceca1d22984b9bae504f054c539dab16406715"} Nov 03 20:10:52.203111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:52.203074 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" podStartSLOduration=2.014003356 podStartE2EDuration="3.203062694s" totalImagesPullingTime="1.189059338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:52.2016163 +0000 UTC m=+555.651722347" watchObservedRunningTime="2026-11-03 20:10:52.203062694 +0000 UTC m=+555.653168739" Nov 03 20:10:52.777047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:52.777007 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:53.189415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:53.189331 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" event={"ID":"669af6be-7305-4087-aafd-16fd9d48a4be","Type":"ContainerStarted","Data":"7d49a7e49bd98f32204f0d673f15d4dbdeb84e636d695a784524c948256ac8e7"} Nov 03 20:10:53.209745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:53.209697 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" podStartSLOduration=1.8560659309999998 podStartE2EDuration="2.209682285s" totalImagesPullingTime="353.616354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:53.208790756 +0000 UTC m=+556.658896802" watchObservedRunningTime="2026-11-03 20:10:53.209682285 +0000 UTC m=+556.659788375" Nov 03 20:10:53.621956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:53.621928 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:10:53.807077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:53.807044 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:54.526083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.526053 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl"] Nov 03 20:10:54.529627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.529609 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.538081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.538059 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 20:10:54.563532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.563509 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl"] Nov 03 20:10:54.630767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.630739 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.630876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.630773 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.630876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.630799 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.630961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.630942 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.630995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.630971 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c8976a-839f-442c-9c08-a0f53663f5d4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.630995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.630990 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgx48\" (UniqueName: \"kubernetes.io/projected/e6c8976a-839f-442c-9c08-a0f53663f5d4-kube-api-access-xgx48\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.676293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.676264 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:54.732345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.732318 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.732525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.732351 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c8976a-839f-442c-9c08-a0f53663f5d4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.732525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.732368 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgx48\" (UniqueName: \"kubernetes.io/projected/e6c8976a-839f-442c-9c08-a0f53663f5d4-kube-api-access-xgx48\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.732525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.732403 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.732525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.732422 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.732525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.732462 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.732813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.732786 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.732910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.732892 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.732967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.732892 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.734538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.734518 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6c8976a-839f-442c-9c08-a0f53663f5d4-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.734875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.734858 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6c8976a-839f-442c-9c08-a0f53663f5d4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.744894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.744869 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgx48\" (UniqueName: \"kubernetes.io/projected/e6c8976a-839f-442c-9c08-a0f53663f5d4-kube-api-access-xgx48\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8rqcl\" (UID: \"e6c8976a-839f-442c-9c08-a0f53663f5d4\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.841604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.841533 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:54.966573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:54.966529 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:55.116048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:55.115974 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl"] Nov 03 20:10:55.118901 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:10:55.118875 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c8976a_839f_442c_9c08_a0f53663f5d4.slice/crio-712903605f07202fb9ffdc09f76bed56e49e4b7a6d1c76e1816f03a6a710edf0 WatchSource:0}: Error finding container 712903605f07202fb9ffdc09f76bed56e49e4b7a6d1c76e1816f03a6a710edf0: Status 404 returned error can't find the container with id 712903605f07202fb9ffdc09f76bed56e49e4b7a6d1c76e1816f03a6a710edf0 Nov 03 20:10:55.199961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:55.199935 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" event={"ID":"e6c8976a-839f-442c-9c08-a0f53663f5d4","Type":"ContainerStarted","Data":"712903605f07202fb9ffdc09f76bed56e49e4b7a6d1c76e1816f03a6a710edf0"} Nov 03 20:10:56.206857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:56.206823 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" event={"ID":"e6c8976a-839f-442c-9c08-a0f53663f5d4","Type":"ContainerStarted","Data":"e4f68f396014c30665a54d7d066dcbaed6e4a8ab2ae07c45bf549222d74a3ee8"} Nov 03 20:10:56.232791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:56.232748 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" podStartSLOduration=1.880105999 podStartE2EDuration="2.232735682s" totalImagesPullingTime="352.629683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:56.231624236 +0000 UTC m=+559.681730282" watchObservedRunningTime="2026-11-03 20:10:56.232735682 +0000 UTC m=+559.682841728" Nov 03 20:10:56.370687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:56.370655 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:56.842185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:56.842146 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:10:58.693936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.693904 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-cjklg"] Nov 03 20:10:58.697435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.697408 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.706578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.706553 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 20:10:58.709305 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.709279 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-cjklg"] Nov 03 20:10:58.773065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.773030 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:10:58.868454 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.868423 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxpvk\" (UniqueName: \"kubernetes.io/projected/6ee62e63-1708-4a53-8f0d-cb6875a87267-kube-api-access-lxpvk\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.868582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.868479 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.868582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.868531 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.868582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.868549 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee62e63-1708-4a53-8f0d-cb6875a87267-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.868691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.868679 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.868728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.868708 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-home\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.969480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.969393 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.969480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.969427 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-home\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.969653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.969486 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpvk\" (UniqueName: \"kubernetes.io/projected/6ee62e63-1708-4a53-8f0d-cb6875a87267-kube-api-access-lxpvk\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.969653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.969503 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.969653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.969516 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.969653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.969532 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee62e63-1708-4a53-8f0d-cb6875a87267-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.969853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.969831 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.969895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.969850 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-home\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.969996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.969977 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.971701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.971678 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ee62e63-1708-4a53-8f0d-cb6875a87267-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.971849 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.971833 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ee62e63-1708-4a53-8f0d-cb6875a87267-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:58.981907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:58.981883 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxpvk\" (UniqueName: \"kubernetes.io/projected/6ee62e63-1708-4a53-8f0d-cb6875a87267-kube-api-access-lxpvk\") pod \"echo-simulator-kserve-64b6f8f96c-cjklg\" (UID: \"6ee62e63-1708-4a53-8f0d-cb6875a87267\") " pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:59.011917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:59.011886 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:10:59.284706 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:59.284677 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-cjklg"] Nov 03 20:10:59.287209 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:10:59.287182 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee62e63_1708_4a53_8f0d_cb6875a87267.slice/crio-5fa4dc91a9990cb1c7df6c3c0a5c9985a8d84e4525519d6ba62cd1e6d7de39a4 WatchSource:0}: Error finding container 5fa4dc91a9990cb1c7df6c3c0a5c9985a8d84e4525519d6ba62cd1e6d7de39a4: Status 404 returned error can't find the container with id 5fa4dc91a9990cb1c7df6c3c0a5c9985a8d84e4525519d6ba62cd1e6d7de39a4 Nov 03 20:10:59.777418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:59.777382 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:59.792532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:59.792503 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:10:59.892191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:10:59.892159 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:00.037756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:00.037626 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:00.230149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:00.230115 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" event={"ID":"6ee62e63-1708-4a53-8f0d-cb6875a87267","Type":"ContainerStarted","Data":"5fa4dc91a9990cb1c7df6c3c0a5c9985a8d84e4525519d6ba62cd1e6d7de39a4"} Nov 03 20:11:00.243884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:00.243847 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8d2rc" Nov 03 20:11:01.235795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:01.235696 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" event={"ID":"6ee62e63-1708-4a53-8f0d-cb6875a87267","Type":"ContainerStarted","Data":"73fa57a457b776fc92c3a95998bb6bfe7d35bb7b78c06d192e09dd67f5cf125b"} Nov 03 20:11:01.259405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:01.259357 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" podStartSLOduration=1.655630567 podStartE2EDuration="3.259344673s" totalImagesPullingTime="1.603714106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:01.256565389 +0000 UTC m=+564.706671445" watchObservedRunningTime="2026-11-03 20:11:01.259344673 +0000 UTC m=+564.709450720" Nov 03 20:11:01.622130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:01.622078 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:11:01.634200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:01.634176 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:11:02.012011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:02.011985 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:11:02.251046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:02.251014 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-lr7kb" Nov 03 20:11:03.016538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.016506 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5"] Nov 03 20:11:03.020143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.020122 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.030534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.030491 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 20:11:03.031934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.031915 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5"] Nov 03 20:11:03.131434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.131367 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:03.213162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.213131 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.213162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.213165 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.213369 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.213320 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.213408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.213363 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6g57\" (UniqueName: \"kubernetes.io/projected/d59d95c7-b4a1-4307-8227-09e1a43822cb-kube-api-access-w6g57\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.213464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.213433 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.213509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.213473 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59d95c7-b4a1-4307-8227-09e1a43822cb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.314390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.314320 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.314390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.314351 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.314907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.314401 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.314907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.314433 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6g57\" (UniqueName: \"kubernetes.io/projected/d59d95c7-b4a1-4307-8227-09e1a43822cb-kube-api-access-w6g57\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.314907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.314484 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.314907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.314509 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59d95c7-b4a1-4307-8227-09e1a43822cb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.314907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.314722 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.314907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.314779 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.314907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.314890 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.316702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.316682 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59d95c7-b4a1-4307-8227-09e1a43822cb-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.316993 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.316973 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59d95c7-b4a1-4307-8227-09e1a43822cb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.326797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.326773 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6g57\" (UniqueName: \"kubernetes.io/projected/d59d95c7-b4a1-4307-8227-09e1a43822cb-kube-api-access-w6g57\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5\" (UID: \"d59d95c7-b4a1-4307-8227-09e1a43822cb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.332989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.332970 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:03.608804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:03.608780 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5"] Nov 03 20:11:03.611074 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:03.611038 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd59d95c7_b4a1_4307_8227_09e1a43822cb.slice/crio-8d4999f3e6983e934926123853ac755d42331bb111397614f057e6c5143a36a0 WatchSource:0}: Error finding container 8d4999f3e6983e934926123853ac755d42331bb111397614f057e6c5143a36a0: Status 404 returned error can't find the container with id 8d4999f3e6983e934926123853ac755d42331bb111397614f057e6c5143a36a0 Nov 03 20:11:04.250850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:04.250761 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" event={"ID":"d59d95c7-b4a1-4307-8227-09e1a43822cb","Type":"ContainerStarted","Data":"9657146a43e73c285ab15b95b1661c2ee390e83906235811457cc34e0438ed2f"} Nov 03 20:11:04.250850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:04.250806 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" event={"ID":"d59d95c7-b4a1-4307-8227-09e1a43822cb","Type":"ContainerStarted","Data":"8d4999f3e6983e934926123853ac755d42331bb111397614f057e6c5143a36a0"} Nov 03 20:11:04.271899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:04.271856 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podStartSLOduration=1.970981193 podStartE2EDuration="2.271844725s" totalImagesPullingTime="300.863532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:04.270308053 +0000 UTC m=+567.720414100" watchObservedRunningTime="2026-11-03 20:11:04.271844725 +0000 UTC m=+567.721950770" Nov 03 20:11:04.333603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:04.333570 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:04.842616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:04.842572 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:11:04.854930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:04.854899 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:11:05.266012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:05.265988 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8rqcl" Nov 03 20:11:09.012775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:09.012736 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:11:09.024773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:09.024743 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:11:09.284336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:09.284264 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-cjklg" Nov 03 20:11:13.333616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:13.333583 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:13.346076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:13.346055 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:14.306581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:14.306552 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:11:17.531086 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:11:17.531047 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:11:21.550482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:21.550391 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:21.680347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:21.680318 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:23.118144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:23.118108 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:23.259085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:23.259046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:28.369345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.369313 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh"] Nov 03 20:11:28.376413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.376393 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.388507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.385462 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:11:28.388507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.387800 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh"] Nov 03 20:11:28.434148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.434114 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.434148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.434146 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.434330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.434218 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.434330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.434251 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.434402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.434358 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dda28c4-26f0-4178-8201-b2cb942ff910-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.434402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.434390 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.434497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.434412 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdrd\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-kube-api-access-8hdrd\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.434537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.434505 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.434590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.434577 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.462585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.462558 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-225mv\"" Nov 03 20:11:28.535746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.535720 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dda28c4-26f0-4178-8201-b2cb942ff910-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.535746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.535750 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.535936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.535769 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdrd\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-kube-api-access-8hdrd\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.535936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.535787 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.535936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.535834 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.535936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.535860 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.535936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.535876 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.536184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.536051 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.536184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.536111 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.536286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.536208 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.536286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.536272 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.536472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.536433 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.536580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.536508 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.536580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.536533 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dda28c4-26f0-4178-8201-b2cb942ff910-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.538208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.538186 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.538416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.538398 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.548133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.548108 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdrd\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-kube-api-access-8hdrd\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.548238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.548152 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-947bh\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.693838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.693762 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:28.973115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:28.973093 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh"] Nov 03 20:11:28.975304 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:28.975272 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dda28c4_26f0_4178_8201_b2cb942ff910.slice/crio-eda9615fecd93d30af0917edee6854f33f06aaf3fce09ef86cee655e5480d50c WatchSource:0}: Error finding container eda9615fecd93d30af0917edee6854f33f06aaf3fce09ef86cee655e5480d50c: Status 404 returned error can't find the container with id eda9615fecd93d30af0917edee6854f33f06aaf3fce09ef86cee655e5480d50c Nov 03 20:11:29.374180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:29.374147 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" event={"ID":"8dda28c4-26f0-4178-8201-b2cb942ff910","Type":"ContainerStarted","Data":"eda9615fecd93d30af0917edee6854f33f06aaf3fce09ef86cee655e5480d50c"} Nov 03 20:11:32.729617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:32.729585 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:32.729891 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:32.729656 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:32.729891 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:32.729689 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:33.396830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:33.396788 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" event={"ID":"8dda28c4-26f0-4178-8201-b2cb942ff910","Type":"ContainerStarted","Data":"e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e"} Nov 03 20:11:33.421827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:33.421781 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podStartSLOduration=1.6701865219999998 podStartE2EDuration="5.421768914s" totalImagesPullingTime="3.751582392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:33.42079766 +0000 UTC m=+596.870903719" watchObservedRunningTime="2026-11-03 20:11:33.421768914 +0000 UTC m=+596.871874959" Nov 03 20:11:33.694257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:33.694171 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:33.695662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:33.695631 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-947bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:11:33.695795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:33.695682 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:11:34.694885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:34.694852 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-947bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:11:34.695363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:34.694921 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:11:35.694930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:35.694893 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-947bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:11:35.695351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:35.694959 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:11:36.694294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:36.694263 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-947bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:11:36.694488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:36.694313 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:11:36.735671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:36.735650 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:11:36.736023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:36.735970 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:11:36.738685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:36.738663 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:11:36.738942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:36.738919 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:11:37.694958 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:37.694914 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-947bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:11:37.695129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:37.694970 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:11:38.694395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:38.694364 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-947bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:11:38.694795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:38.694414 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:11:39.695122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:39.695083 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-947bh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 20:11:39.695510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:39.695161 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 20:11:41.045397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:41.045351 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:41.045889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:41.045797 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:41.046761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:41.046738 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 20:11:43.364318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.364279 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5"] Nov 03 20:11:43.369117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.369088 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.382164 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.382134 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5"] Nov 03 20:11:43.397130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.397095 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td"] Nov 03 20:11:43.401887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.401863 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.415715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.415675 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td"] Nov 03 20:11:43.487860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.487822 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56f20a7-a8f7-4aed-86c2-28929da807ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.487860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.487864 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.488076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.487883 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.488076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488018 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a732ad93-52e7-436f-94b7-03ddd72075da-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.488076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488050 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.488171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488075 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a732ad93-52e7-436f-94b7-03ddd72075da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.488171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488107 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.488233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488176 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.488233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488207 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a732ad93-52e7-436f-94b7-03ddd72075da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.488289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488234 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.488289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488271 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59ct\" (UniqueName: \"kubernetes.io/projected/a732ad93-52e7-436f-94b7-03ddd72075da-kube-api-access-c59ct\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.488353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488290 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxkp8\" (UniqueName: \"kubernetes.io/projected/d56f20a7-a8f7-4aed-86c2-28929da807ee-kube-api-access-jxkp8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.488353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488312 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.488411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488368 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.488462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488424 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.488502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488459 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.488502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488483 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.488565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.488501 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.589696 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589659 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a732ad93-52e7-436f-94b7-03ddd72075da-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.589696 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589702 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.589925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589719 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a732ad93-52e7-436f-94b7-03ddd72075da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.589925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589737 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.589925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589761 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.589925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589791 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a732ad93-52e7-436f-94b7-03ddd72075da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.589925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589823 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.589925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589849 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59ct\" (UniqueName: \"kubernetes.io/projected/a732ad93-52e7-436f-94b7-03ddd72075da-kube-api-access-c59ct\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.589925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589876 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxkp8\" (UniqueName: \"kubernetes.io/projected/d56f20a7-a8f7-4aed-86c2-28929da807ee-kube-api-access-jxkp8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.589925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589903 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.590441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.589967 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.590441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590120 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.590441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590160 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.590441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590204 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.590441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590243 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.590441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590277 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56f20a7-a8f7-4aed-86c2-28929da807ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.590441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590283 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.590441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590303 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.590441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590320 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.590988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590566 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.590988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590592 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.590988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590667 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.590988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.590883 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.591195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.591153 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.591251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.591202 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56f20a7-a8f7-4aed-86c2-28929da807ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.591321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.591292 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a732ad93-52e7-436f-94b7-03ddd72075da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.591391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.591371 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.591484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.591465 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.592431 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.592406 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.592844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.592826 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a732ad93-52e7-436f-94b7-03ddd72075da-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.592967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.592946 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a732ad93-52e7-436f-94b7-03ddd72075da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.593009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.592954 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.602315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.602266 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56f20a7-a8f7-4aed-86c2-28929da807ee-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.602436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.602340 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59ct\" (UniqueName: \"kubernetes.io/projected/a732ad93-52e7-436f-94b7-03ddd72075da-kube-api-access-c59ct\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.602436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.602388 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a732ad93-52e7-436f-94b7-03ddd72075da-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-jbqn5\" (UID: \"a732ad93-52e7-436f-94b7-03ddd72075da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.602958 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.602927 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxkp8\" (UniqueName: \"kubernetes.io/projected/d56f20a7-a8f7-4aed-86c2-28929da807ee-kube-api-access-jxkp8\") pod \"maas-default-gateway-openshift-default-59f957fdd7-qq2td\" (UID: \"d56f20a7-a8f7-4aed-86c2-28929da807ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.686507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.686419 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:43.721913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.721868 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:43.971943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.971884 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5"] Nov 03 20:11:43.974497 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:43.974466 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda732ad93_52e7_436f_94b7_03ddd72075da.slice/crio-e1c957534e64284a8f79c58a738cc0b5dbaab5b5552e851b7a16347ac02a1640 WatchSource:0}: Error finding container e1c957534e64284a8f79c58a738cc0b5dbaab5b5552e851b7a16347ac02a1640: Status 404 returned error can't find the container with id e1c957534e64284a8f79c58a738cc0b5dbaab5b5552e851b7a16347ac02a1640 Nov 03 20:11:43.976656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.976617 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:43.976752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.976697 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:43.976817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:43.976747 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:44.015335 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:44.015304 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56f20a7_a8f7_4aed_86c2_28929da807ee.slice/crio-3a674a3146bd629c07d0bfcc78f4ab86b9058b9f67b79cc25729d936e76cc6e9 WatchSource:0}: Error finding container 3a674a3146bd629c07d0bfcc78f4ab86b9058b9f67b79cc25729d936e76cc6e9: Status 404 returned error can't find the container with id 3a674a3146bd629c07d0bfcc78f4ab86b9058b9f67b79cc25729d936e76cc6e9 Nov 03 20:11:44.016877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.016848 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td"] Nov 03 20:11:44.021301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.021269 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:44.021365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.021338 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:44.021405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.021367 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:44.461880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.461838 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" event={"ID":"d56f20a7-a8f7-4aed-86c2-28929da807ee","Type":"ContainerStarted","Data":"913dc353eddabbe29e614546ee4a049ca1c1e3edbe84577d4bb5a445b1dbd7e4"} Nov 03 20:11:44.462456 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.461907 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" event={"ID":"d56f20a7-a8f7-4aed-86c2-28929da807ee","Type":"ContainerStarted","Data":"3a674a3146bd629c07d0bfcc78f4ab86b9058b9f67b79cc25729d936e76cc6e9"} Nov 03 20:11:44.463589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.463562 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" event={"ID":"a732ad93-52e7-436f-94b7-03ddd72075da","Type":"ContainerStarted","Data":"d72ddf93e6c4956895ab2916079130438643008150c4cfdf0eec65bc17e17301"} Nov 03 20:11:44.463698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.463614 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" event={"ID":"a732ad93-52e7-436f-94b7-03ddd72075da","Type":"ContainerStarted","Data":"e1c957534e64284a8f79c58a738cc0b5dbaab5b5552e851b7a16347ac02a1640"} Nov 03 20:11:44.496469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.496184 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podStartSLOduration=1.496128442 podStartE2EDuration="1.496128442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:44.484842216 +0000 UTC m=+607.934948263" watchObservedRunningTime="2026-11-03 20:11:44.496128442 +0000 UTC m=+607.946234489" Nov 03 20:11:44.521148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.521085 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podStartSLOduration=1.521062125 podStartE2EDuration="1.521062125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:44.519401068 +0000 UTC m=+607.969507194" watchObservedRunningTime="2026-11-03 20:11:44.521062125 +0000 UTC m=+607.971168171" Nov 03 20:11:44.696713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.696680 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:44.722422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:44.722325 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:45.694384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:45.694340 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:11:45.695018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:45.694985 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:11:45.724102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:45.724066 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:11:45.724280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:45.724120 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:11:46.706472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:46.705571 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:11:46.706472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:46.705636 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:11:46.723085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:46.723047 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:11:46.723254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:46.723095 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:11:47.545578 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:11:47.545543 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:11:47.688470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:47.687877 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:11:47.688470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:47.687941 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:11:47.723204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:47.723163 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:11:47.723617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:47.723216 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:11:48.700470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:48.699884 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:11:48.700470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:48.699944 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:11:48.722771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:48.722720 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:11:48.722771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:48.722771 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:11:49.606412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:49.606377 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:49.687861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:49.687822 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:11:49.688136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:49.688107 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:11:49.722669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:49.722627 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:11:49.722833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:49.722691 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:11:49.762826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:49.762791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:49.914638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:49.914545 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:50.139617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.139580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:50.242736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.242062 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2"] Nov 03 20:11:50.247944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.247920 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.258415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.258297 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e17fc195-openshift-default-dockercfg-9g6x9\"" Nov 03 20:11:50.265748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.265720 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2"] Nov 03 20:11:50.272438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.272366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:50.375190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.375155 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-envoy\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.375371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.375194 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istiod-ca-cert\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.375371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.375223 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-token\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.375476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.375385 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4rn\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-kube-api-access-7c4rn\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.375533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.375490 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-credential-socket\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.375577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.375556 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-socket\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.375615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.375599 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-certs\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.375710 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.375693 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-podinfo\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.375760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.375745 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-data\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.410744 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.410707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:50.476614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.476575 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-credential-socket\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.476811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.476629 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-socket\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.476811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.476764 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-certs\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.476936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.476843 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-podinfo\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.476936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.476890 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-data\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.477035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.476941 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-envoy\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.477035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.476974 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istiod-ca-cert\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.477035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.477004 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-token\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.477185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.477035 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-socket\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.477185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.477054 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4rn\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-kube-api-access-7c4rn\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.477185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.477097 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-credential-socket\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.477425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.477402 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-data\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.477650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.477630 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-certs\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.477923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.477900 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istiod-ca-cert\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.479655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.479623 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-podinfo\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.480551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.480531 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-envoy\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.495335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.495269 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4rn\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-kube-api-access-7c4rn\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.502343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.502316 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-token\") pod \"e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.566304 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.566238 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:50.571167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.571140 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:50.686739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.686706 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:11:50.687085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.686746 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:11:50.707414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.707377 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:50.722971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.722937 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 20:11:50.723124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.723010 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 20:11:50.834625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.834587 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:50.866224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.866198 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2"] Nov 03 20:11:50.868556 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:50.868530 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba71fb4e_7cb0_4d13_a749_3548bed0e4d8.slice/crio-cc232d798b20ad95d340e8814b655ac624f8785ee057ae15ad4e34a6f0808cda WatchSource:0}: Error finding container cc232d798b20ad95d340e8814b655ac624f8785ee057ae15ad4e34a6f0808cda: Status 404 returned error can't find the container with id cc232d798b20ad95d340e8814b655ac624f8785ee057ae15ad4e34a6f0808cda Nov 03 20:11:50.871559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.871521 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:50.871641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.871608 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:50.871680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:50.871656 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:51.522626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.522585 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:51.547324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.547290 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" event={"ID":"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8","Type":"ContainerStarted","Data":"42c5b20882c6ed087ed5954b743a149127d0aa210b98ce8aca563180104fb055"} Nov 03 20:11:51.547520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.547352 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" event={"ID":"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8","Type":"ContainerStarted","Data":"cc232d798b20ad95d340e8814b655ac624f8785ee057ae15ad4e34a6f0808cda"} Nov 03 20:11:51.571774 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.571741 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:51.576167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.576111 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" podStartSLOduration=1.576093873 podStartE2EDuration="1.576093873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:51.573593853 +0000 UTC m=+615.023699900" watchObservedRunningTime="2026-11-03 20:11:51.576093873 +0000 UTC m=+615.026199920" Nov 03 20:11:51.578405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.578382 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:51.687075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.687042 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 20:11:51.687481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.687090 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 20:11:51.901426 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.901334 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc"] Nov 03 20:11:51.905959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.905933 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:51.922646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.922293 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc"] Nov 03 20:11:51.943070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.943015 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4bd752fd-openshift-default-dockercfg-g95s7\"" Nov 03 20:11:51.983169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:51.983120 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:52.000968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.000931 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304d6315-c67b-457d-ba22-f746bf08a957-istiod-ca-cert\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.001146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.000986 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-socket\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.001146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.001082 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304d6315-c67b-457d-ba22-f746bf08a957-istio-podinfo\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.001278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.001185 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-data\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.001278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.001234 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-credential-socket\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.001278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.001272 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5kl\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-kube-api-access-5x5kl\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.001463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.001317 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-certs\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.001463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.001361 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-istio-token\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.001463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.001407 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-envoy\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.102534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.102483 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5kl\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-kube-api-access-5x5kl\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.102732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.102561 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-certs\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.102732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.102611 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-istio-token\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.102732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.102662 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-envoy\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.102732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.102723 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304d6315-c67b-457d-ba22-f746bf08a957-istiod-ca-cert\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.102962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.102762 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-socket\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.102962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.102795 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304d6315-c67b-457d-ba22-f746bf08a957-istio-podinfo\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.102962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.102869 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-data\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.102962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.102917 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-credential-socket\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.103729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.103412 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-credential-socket\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.104173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.104157 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304d6315-c67b-457d-ba22-f746bf08a957-istiod-ca-cert\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.104924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.104460 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-data\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.104924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.104649 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-certs\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.106638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.106614 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-socket\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.116634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.116576 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-envoy\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.123358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.122571 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304d6315-c67b-457d-ba22-f746bf08a957-istio-podinfo\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.129232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.129078 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:52.130265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.130214 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-istio-token\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.130556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.130532 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5kl\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-kube-api-access-5x5kl\") pod \"e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.228947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.228848 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:52.543220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.543188 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc"] Nov 03 20:11:52.550422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.550374 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:52.550691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.550664 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:52.550864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.550835 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:52.558900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.558872 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" event={"ID":"304d6315-c67b-457d-ba22-f746bf08a957","Type":"ContainerStarted","Data":"dc2b5366aac79ba39ab2b557e5ca555fd329e889c406a50bff8f01229409b738"} Nov 03 20:11:52.559480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.559301 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:52.560526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.560504 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:11:52.723590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.723545 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:11:52.724036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.723610 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:11:52.759518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.759384 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:52.759860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.759796 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:52.760771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.760750 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" Nov 03 20:11:52.773076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.773037 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:52.773794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.773773 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:52.775602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:52.775586 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" Nov 03 20:11:53.499296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:53.499260 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:53.565024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:53.564984 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" event={"ID":"304d6315-c67b-457d-ba22-f746bf08a957","Type":"ContainerStarted","Data":"11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1"} Nov 03 20:11:53.605710 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:53.605666 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" podStartSLOduration=2.605653141 podStartE2EDuration="2.605653141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:53.603734564 +0000 UTC m=+617.053840625" watchObservedRunningTime="2026-11-03 20:11:53.605653141 +0000 UTC m=+617.055759186" Nov 03 20:11:53.633988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:53.633917 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:53.764501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:53.764469 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:53.975686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:53.975636 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:54.111515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.111384 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:54.229508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.229477 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:54.234305 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.234278 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:54.245400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.245372 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:54.249491 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.249470 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8"] Nov 03 20:11:54.253141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.253125 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:11:54.262757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.262737 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-459ca04a\"" Nov 03 20:11:54.262757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.263614 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-459ca04a-dockercfg-mxgds\"" Nov 03 20:11:54.267682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.266830 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8"] Nov 03 20:11:54.287064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.287037 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj"] Nov 03 20:11:54.290653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.290632 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:11:54.302521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.302493 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj"] Nov 03 20:11:54.335061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.335037 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5tb\" (UniqueName: \"kubernetes.io/projected/5ce0fb41-a836-469b-ac55-84c448c650b1-kube-api-access-8p5tb\") pod \"payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8\" (UID: \"5ce0fb41-a836-469b-ac55-84c448c650b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:11:54.335184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.335096 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce0fb41-a836-469b-ac55-84c448c650b1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8\" (UID: \"5ce0fb41-a836-469b-ac55-84c448c650b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:11:54.376016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.375946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:54.435798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.435765 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj\" (UID: \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:11:54.435974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.435809 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce0fb41-a836-469b-ac55-84c448c650b1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8\" (UID: \"5ce0fb41-a836-469b-ac55-84c448c650b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:11:54.436026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.435988 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68g8\" (UniqueName: \"kubernetes.io/projected/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-kube-api-access-b68g8\") pod \"payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj\" (UID: \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:11:54.436077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.436026 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5tb\" (UniqueName: \"kubernetes.io/projected/5ce0fb41-a836-469b-ac55-84c448c650b1-kube-api-access-8p5tb\") pod \"payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8\" (UID: \"5ce0fb41-a836-469b-ac55-84c448c650b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:11:54.436543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.436517 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce0fb41-a836-469b-ac55-84c448c650b1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8\" (UID: \"5ce0fb41-a836-469b-ac55-84c448c650b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:11:54.457245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.457223 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5tb\" (UniqueName: \"kubernetes.io/projected/5ce0fb41-a836-469b-ac55-84c448c650b1-kube-api-access-8p5tb\") pod \"payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8\" (UID: \"5ce0fb41-a836-469b-ac55-84c448c650b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:11:54.537672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.537640 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68g8\" (UniqueName: \"kubernetes.io/projected/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-kube-api-access-b68g8\") pod \"payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj\" (UID: \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:11:54.537876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.537743 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj\" (UID: \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:11:54.538681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.538657 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj\" (UID: \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:11:54.549820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.549794 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68g8\" (UniqueName: \"kubernetes.io/projected/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-kube-api-access-b68g8\") pod \"payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj\" (UID: \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:11:54.569974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.569936 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:11:54.570401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.570381 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:54.571184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.571168 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:11:54.607887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.607669 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:11:54.899919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.899847 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8"] Nov 03 20:11:54.904099 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:54.904071 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce0fb41_a836_469b_ac55_84c448c650b1.slice/crio-329b5d99d0e47aad3139355ca57ae9656f264e97ff30a95a5535077d9b67bb63 WatchSource:0}: Error finding container 329b5d99d0e47aad3139355ca57ae9656f264e97ff30a95a5535077d9b67bb63: Status 404 returned error can't find the container with id 329b5d99d0e47aad3139355ca57ae9656f264e97ff30a95a5535077d9b67bb63 Nov 03 20:11:54.938627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:54.938602 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj"] Nov 03 20:11:55.003180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.003147 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:55.080794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.080758 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh"] Nov 03 20:11:55.084522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.084494 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:11:55.087665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.087635 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh"] Nov 03 20:11:55.088646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.088620 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e17fc195\"" Nov 03 20:11:55.147149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.147118 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:55.160916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.160852 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e17fc195-dockercfg-gvbw4\"" Nov 03 20:11:55.246529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.246479 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-plugins-config-volume\") pod \"payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh\" (UID: \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:11:55.246729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.246703 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2hz\" (UniqueName: \"kubernetes.io/projected/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-kube-api-access-mq2hz\") pod \"payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh\" (UID: \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:11:55.347474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.347412 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-plugins-config-volume\") pod \"payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh\" (UID: \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:11:55.347660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.347573 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2hz\" (UniqueName: \"kubernetes.io/projected/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-kube-api-access-mq2hz\") pod \"payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh\" (UID: \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:11:55.348195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.348172 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-plugins-config-volume\") pod \"payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh\" (UID: \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:11:55.353026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.352988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:55.360264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.360239 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2hz\" (UniqueName: \"kubernetes.io/projected/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-kube-api-access-mq2hz\") pod \"payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh\" (UID: \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:11:55.398181 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.398149 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:11:55.486722 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.486680 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:55.577458 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.577400 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" event={"ID":"59119cbf-9835-4eb1-a44d-1fb4e30e97bd","Type":"ContainerStarted","Data":"409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327"} Nov 03 20:11:55.577651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.577475 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" event={"ID":"59119cbf-9835-4eb1-a44d-1fb4e30e97bd","Type":"ContainerStarted","Data":"770c9b1c5b2d1fe93ef58e5cc0c889b2513759a54bddfdb11f2d59241460d668"} Nov 03 20:11:55.577651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.577504 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:11:55.579105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.579079 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" event={"ID":"5ce0fb41-a836-469b-ac55-84c448c650b1","Type":"ContainerStarted","Data":"4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e"} Nov 03 20:11:55.579228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.579131 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" event={"ID":"5ce0fb41-a836-469b-ac55-84c448c650b1","Type":"ContainerStarted","Data":"329b5d99d0e47aad3139355ca57ae9656f264e97ff30a95a5535077d9b67bb63"} Nov 03 20:11:55.579294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.579234 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:11:55.630722 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.630668 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" podStartSLOduration=1.630651503 podStartE2EDuration="1.630651503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:55.628309793 +0000 UTC m=+619.078415837" watchObservedRunningTime="2026-11-03 20:11:55.630651503 +0000 UTC m=+619.080757550" Nov 03 20:11:55.630876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.630776 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" podStartSLOduration=1.630769406 podStartE2EDuration="1.630769406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:55.607504376 +0000 UTC m=+619.057610421" watchObservedRunningTime="2026-11-03 20:11:55.630769406 +0000 UTC m=+619.080875452" Nov 03 20:11:55.672503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.672470 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:55.679354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.679335 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh"] Nov 03 20:11:55.681427 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:55.681401 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc97aa6_a8f7_43d7_a9a2_ebb256750cc8.slice/crio-10dc98c6a4d184a09d14876e65777da6d9dcd5b2635d03ac26a660a5ee1e4f9a WatchSource:0}: Error finding container 10dc98c6a4d184a09d14876e65777da6d9dcd5b2635d03ac26a660a5ee1e4f9a: Status 404 returned error can't find the container with id 10dc98c6a4d184a09d14876e65777da6d9dcd5b2635d03ac26a660a5ee1e4f9a Nov 03 20:11:55.698229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.698176 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l"] Nov 03 20:11:55.703045 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.703024 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:11:55.710250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.710225 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9dd42e13-dockercfg-r9zmp\"" Nov 03 20:11:55.713030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.712775 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l"] Nov 03 20:11:55.717611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.717547 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9dd42e13\"" Nov 03 20:11:55.807325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.807290 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:55.853777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.853745 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5acad-67c8-4370-8d45-36117dd8b248-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l\" (UID: \"dff5acad-67c8-4370-8d45-36117dd8b248\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:11:55.853930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.853809 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zxn\" (UniqueName: \"kubernetes.io/projected/dff5acad-67c8-4370-8d45-36117dd8b248-kube-api-access-b9zxn\") pod \"payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l\" (UID: \"dff5acad-67c8-4370-8d45-36117dd8b248\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:11:55.954987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.954903 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zxn\" (UniqueName: \"kubernetes.io/projected/dff5acad-67c8-4370-8d45-36117dd8b248-kube-api-access-b9zxn\") pod \"payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l\" (UID: \"dff5acad-67c8-4370-8d45-36117dd8b248\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:11:55.955412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.955025 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5acad-67c8-4370-8d45-36117dd8b248-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l\" (UID: \"dff5acad-67c8-4370-8d45-36117dd8b248\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:11:55.955809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.955788 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5acad-67c8-4370-8d45-36117dd8b248-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l\" (UID: \"dff5acad-67c8-4370-8d45-36117dd8b248\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:11:55.967205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:55.967176 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zxn\" (UniqueName: \"kubernetes.io/projected/dff5acad-67c8-4370-8d45-36117dd8b248-kube-api-access-b9zxn\") pod \"payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l\" (UID: \"dff5acad-67c8-4370-8d45-36117dd8b248\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:11:56.028302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.028270 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:11:56.311476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.311418 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l"] Nov 03 20:11:56.311923 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:56.311892 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff5acad_67c8_4370_8d45_36117dd8b248.slice/crio-212a9a08cea4d88ae6a8ff40a55d04e27a557e884e8502e511e4c9040e24bb7a WatchSource:0}: Error finding container 212a9a08cea4d88ae6a8ff40a55d04e27a557e884e8502e511e4c9040e24bb7a: Status 404 returned error can't find the container with id 212a9a08cea4d88ae6a8ff40a55d04e27a557e884e8502e511e4c9040e24bb7a Nov 03 20:11:56.499171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.499137 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:56.586476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.586359 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" event={"ID":"dff5acad-67c8-4370-8d45-36117dd8b248","Type":"ContainerStarted","Data":"be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611"} Nov 03 20:11:56.586476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.586438 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" event={"ID":"dff5acad-67c8-4370-8d45-36117dd8b248","Type":"ContainerStarted","Data":"212a9a08cea4d88ae6a8ff40a55d04e27a557e884e8502e511e4c9040e24bb7a"} Nov 03 20:11:56.586700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.586525 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:11:56.588579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.588548 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" event={"ID":"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8","Type":"ContainerStarted","Data":"6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60"} Nov 03 20:11:56.588734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.588599 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" event={"ID":"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8","Type":"ContainerStarted","Data":"10dc98c6a4d184a09d14876e65777da6d9dcd5b2635d03ac26a660a5ee1e4f9a"} Nov 03 20:11:56.589000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.588983 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:11:56.613839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.613791 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" podStartSLOduration=1.6137791510000001 podStartE2EDuration="1.613779151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:56.613604428 +0000 UTC m=+620.063710474" watchObservedRunningTime="2026-11-03 20:11:56.613779151 +0000 UTC m=+620.063885197" Nov 03 20:11:56.637274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.637226 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" podStartSLOduration=1.6372097559999998 podStartE2EDuration="1.637209756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:56.636197749 +0000 UTC m=+620.086303802" watchObservedRunningTime="2026-11-03 20:11:56.637209756 +0000 UTC m=+620.087315802" Nov 03 20:11:56.704213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.704167 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:56.796019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.795975 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm"] Nov 03 20:11:56.799885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.799865 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:11:56.809500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.809478 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm"] Nov 03 20:11:56.838327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.838231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:56.864794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.864760 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4bd752fd\"" Nov 03 20:11:56.891132 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.891099 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4bd752fd-dockercfg-6f27b\"" Nov 03 20:11:56.969089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.969053 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm\" (UID: \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:11:56.969502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:56.969246 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-kube-api-access-lnmn7\") pod \"payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm\" (UID: \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:11:57.070599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.070560 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-kube-api-access-lnmn7\") pod \"payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm\" (UID: \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:11:57.070741 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.070611 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm\" (UID: \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:11:57.071346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.071326 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm\" (UID: \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:11:57.084948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.084915 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-kube-api-access-lnmn7\") pod \"payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm\" (UID: \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:11:57.114522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.114463 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:11:57.390352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.390328 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm"] Nov 03 20:11:57.392206 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:57.392177 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e86b7ec_ee6a_44ea_a6e7_9acb448e03ed.slice/crio-65d50adc8971e16587488d4e6f40c743db69824ba26e0b74806b72b2842e1c98 WatchSource:0}: Error finding container 65d50adc8971e16587488d4e6f40c743db69824ba26e0b74806b72b2842e1c98: Status 404 returned error can't find the container with id 65d50adc8971e16587488d4e6f40c743db69824ba26e0b74806b72b2842e1c98 Nov 03 20:11:57.539713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.539676 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb"] Nov 03 20:11:57.544288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.544264 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.548534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.548505 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:11:57.548685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.548648 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e17fc195-serving-cert\"" Nov 03 20:11:57.548685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.548660 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:11:57.550096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.550073 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqpwn\"" Nov 03 20:11:57.550204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.550137 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:11:57.553183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.553161 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb"] Nov 03 20:11:57.595848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.595814 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" event={"ID":"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed","Type":"ContainerStarted","Data":"5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933"} Nov 03 20:11:57.596045 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.595874 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" event={"ID":"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed","Type":"ContainerStarted","Data":"65d50adc8971e16587488d4e6f40c743db69824ba26e0b74806b72b2842e1c98"} Nov 03 20:11:57.618940 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.618884 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" podStartSLOduration=1.618871585 podStartE2EDuration="1.618871585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:57.618729035 +0000 UTC m=+621.068835084" watchObservedRunningTime="2026-11-03 20:11:57.618871585 +0000 UTC m=+621.068977629" Nov 03 20:11:57.643187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.643105 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:57.679219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.679189 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-openshift-service-ca\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.679399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.679376 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqwp\" (UniqueName: \"kubernetes.io/projected/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-kube-api-access-gkqwp\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.679576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.679416 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-maas-api-tls\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.679728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.679709 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-metrics-certs\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.781222 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.781175 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-metrics-certs\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.781402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.781238 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-openshift-service-ca\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.781402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.781324 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqwp\" (UniqueName: \"kubernetes.io/projected/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-kube-api-access-gkqwp\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.781402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.781353 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-maas-api-tls\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.782018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.781990 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-openshift-service-ca\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.783829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.783805 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-maas-api-tls\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.783933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.783804 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-metrics-certs\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.784709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.784672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:57.793793 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.793771 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqwp\" (UniqueName: \"kubernetes.io/projected/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-kube-api-access-gkqwp\") pod \"maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:57.860820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:57.860788 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:58.141060 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:58.141029 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb"] Nov 03 20:11:58.142717 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:11:58.142691 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18286a56_9ac9_4ff2_88e0_9a2c7a91e216.slice/crio-e7e5137b391c3f7c78a50c633665fc969dcb714c75f35f13d188ebda136a85a3 WatchSource:0}: Error finding container e7e5137b391c3f7c78a50c633665fc969dcb714c75f35f13d188ebda136a85a3: Status 404 returned error can't find the container with id e7e5137b391c3f7c78a50c633665fc969dcb714c75f35f13d188ebda136a85a3 Nov 03 20:11:58.144182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:58.144165 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:11:58.365410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:58.365365 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:58.564758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:58.564684 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:58.609556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:58.609517 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" event={"ID":"18286a56-9ac9-4ff2-88e0-9a2c7a91e216","Type":"ContainerStarted","Data":"e7e5137b391c3f7c78a50c633665fc969dcb714c75f35f13d188ebda136a85a3"} Nov 03 20:11:58.609737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:58.609630 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:11:58.708724 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:58.708687 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:58.860946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:58.860910 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:59.179091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:59.179049 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:59.313114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:59.313059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:59.601245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:59.601207 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:11:59.615712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:59.615682 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" event={"ID":"18286a56-9ac9-4ff2-88e0-9a2c7a91e216","Type":"ContainerStarted","Data":"bdb0eeb4f7d4e507cb81d97a3450acf68de5713d40ca269fd1451d20dc838435"} Nov 03 20:11:59.615952 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:59.615931 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:11:59.647470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:11:59.640652 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" podStartSLOduration=2.20369564 podStartE2EDuration="2.64063107s" totalImagesPullingTime="436.93543ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:59.637744078 +0000 UTC m=+623.087850137" watchObservedRunningTime="2026-11-03 20:11:59.64063107 +0000 UTC m=+623.090737117" Nov 03 20:12:00.455679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:00.455646 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:01.450893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.450854 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2"] Nov 03 20:12:01.455794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.455766 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.466008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.465988 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4bd752fd-serving-cert\"" Nov 03 20:12:01.467158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.467124 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2"] Nov 03 20:12:01.535554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.535514 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:01.621904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.621865 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-maas-api-tls\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.622052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.622028 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8th2x\" (UniqueName: \"kubernetes.io/projected/823460b4-3334-4436-9dc0-df5f5e4a323a-kube-api-access-8th2x\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.622125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.622087 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-metrics-certs\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.622181 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.622166 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/823460b4-3334-4436-9dc0-df5f5e4a323a-openshift-service-ca\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.668829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.668796 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:01.723468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.723360 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-maas-api-tls\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.723625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.723482 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8th2x\" (UniqueName: \"kubernetes.io/projected/823460b4-3334-4436-9dc0-df5f5e4a323a-kube-api-access-8th2x\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.723625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.723527 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-metrics-certs\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.723625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.723594 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/823460b4-3334-4436-9dc0-df5f5e4a323a-openshift-service-ca\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.724373 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.724350 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/823460b4-3334-4436-9dc0-df5f5e4a323a-openshift-service-ca\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.726064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.726034 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-metrics-certs\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.726290 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.726269 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-maas-api-tls\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.735619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.735595 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8th2x\" (UniqueName: \"kubernetes.io/projected/823460b4-3334-4436-9dc0-df5f5e4a323a-kube-api-access-8th2x\") pod \"maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.772492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.772436 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:01.925425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:01.925390 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:02.057608 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:02.057572 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:02.058104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:02.058085 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2"] Nov 03 20:12:02.060773 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:02.060744 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823460b4_3334_4436_9dc0_df5f5e4a323a.slice/crio-6d497a2d40b39800aaec3bcebc3ac931cf04ec3e1209889db7ffc70b7c808768 WatchSource:0}: Error finding container 6d497a2d40b39800aaec3bcebc3ac931cf04ec3e1209889db7ffc70b7c808768: Status 404 returned error can't find the container with id 6d497a2d40b39800aaec3bcebc3ac931cf04ec3e1209889db7ffc70b7c808768 Nov 03 20:12:02.632240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:02.632204 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" event={"ID":"823460b4-3334-4436-9dc0-df5f5e4a323a","Type":"ContainerStarted","Data":"896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352"} Nov 03 20:12:02.632635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:02.632254 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" event={"ID":"823460b4-3334-4436-9dc0-df5f5e4a323a","Type":"ContainerStarted","Data":"6d497a2d40b39800aaec3bcebc3ac931cf04ec3e1209889db7ffc70b7c808768"} Nov 03 20:12:02.632635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:02.632382 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:02.656181 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:02.656141 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" podStartSLOduration=1.203220578 podStartE2EDuration="1.656126551s" totalImagesPullingTime="452.905973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:02.654795802 +0000 UTC m=+626.104901846" watchObservedRunningTime="2026-11-03 20:12:02.656126551 +0000 UTC m=+626.106232596" Nov 03 20:12:02.659585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:02.659557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:03.145414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:03.145376 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:03.323945 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:03.323896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:03.623625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:03.623571 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:04.323702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:04.323670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:05.113250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:05.113214 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:05.626328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:05.626297 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:12:05.959088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:05.958990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:06.094260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:06.094222 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:06.593504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:06.593472 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:12:06.593686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:06.593660 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:12:07.027938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:07.027911 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:07.599278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:07.599247 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:12:07.599486 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:07.599415 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:12:07.834936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:07.834900 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:08.600611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:08.600570 4711 patch_prober.go:29] interesting pod/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:12:08.601007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:08.600633 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:08.645481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:08.645417 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:08.916161 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:08.916066 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:09.619540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:09.619503 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:12:09.697510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:09.697476 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:09.830311 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:09.830237 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:10.107556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:10.107520 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:11.087251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:11.087165 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:11.713923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:11.713892 4711 generic.go:356] "Generic (PLEG): container finished" podID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerID="42c5b20882c6ed087ed5954b743a149127d0aa210b98ce8aca563180104fb055" exitCode=0 Nov 03 20:12:11.714132 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:11.713943 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" event={"ID":"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8","Type":"ContainerDied","Data":"42c5b20882c6ed087ed5954b743a149127d0aa210b98ce8aca563180104fb055"} Nov 03 20:12:11.714311 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:11.714298 4711 scope.go:122] "RemoveContainer" containerID="42c5b20882c6ed087ed5954b743a149127d0aa210b98ce8aca563180104fb055" Nov 03 20:12:11.715596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:11.715557 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:11.715669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:11.715655 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:11.715708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:11.715691 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:11.884462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:11.884360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:12.621679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:12.621646 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:12.721910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:12.721877 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" event={"ID":"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8","Type":"ContainerStarted","Data":"248c09fce515b17822a68f1970a3e43159a42041bbd892b77bcfdf1cd1a25494"} Nov 03 20:12:13.515092 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:13.515058 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:13.585470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:13.580837 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:12:13.585470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:13.580963 4711 patch_prober.go:29] interesting pod/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:12:13.585470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:13.581005 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:12:14.503662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:14.503622 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:14.582884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:14.582842 4711 patch_prober.go:29] interesting pod/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:12:14.583146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:14.583116 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:12:15.556106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:15.556068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:15.575498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:15.572818 4711 patch_prober.go:29] interesting pod/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:12:15.575498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:15.572877 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:12:16.581032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:16.580817 4711 patch_prober.go:29] interesting pod/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:12:16.581607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:16.581569 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:12:16.621828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:16.621796 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:17.534220 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:17.534178 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:12:17.572454 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:17.572401 4711 patch_prober.go:29] interesting pod/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 03 20:12:17.572613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:17.572487 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 03 20:12:17.755325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:17.755297 4711 generic.go:356] "Generic (PLEG): container finished" podID="304d6315-c67b-457d-ba22-f746bf08a957" containerID="11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1" exitCode=0 Nov 03 20:12:17.755907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:17.755369 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" event={"ID":"304d6315-c67b-457d-ba22-f746bf08a957","Type":"ContainerDied","Data":"11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1"} Nov 03 20:12:17.755907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:17.755770 4711 scope.go:122] "RemoveContainer" containerID="11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1" Nov 03 20:12:17.757004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:17.756977 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:17.757086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:17.757041 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:17.757086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:17.757070 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:17.771325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:17.771295 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:18.595787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:18.595752 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:12:18.597368 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:18.597345 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:12:18.603489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:18.603466 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:12:18.762379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:18.762339 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" event={"ID":"304d6315-c67b-457d-ba22-f746bf08a957","Type":"ContainerStarted","Data":"252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba"} Nov 03 20:12:18.950270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:18.950188 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8"] Nov 03 20:12:18.962464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:18.962423 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj"] Nov 03 20:12:18.999108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:18.999044 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:19.145242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:19.145201 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:19.229509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:19.229397 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:12:19.229689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:19.229636 4711 patch_prober.go:29] interesting pod/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 20:12:19.229772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:19.229697 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 20:12:20.028548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:20.028517 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:20.230105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:20.230070 4711 patch_prober.go:29] interesting pod/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 20:12:20.230280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:20.230127 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 20:12:21.026979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:21.026933 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:21.229838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:21.229799 4711 patch_prober.go:29] interesting pod/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 20:12:21.230258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:21.229874 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 20:12:21.897731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:21.897671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:22.229197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:22.229122 4711 patch_prober.go:29] interesting pod/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 20:12:22.229197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:22.229167 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 20:12:22.591895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:22.591850 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:22.757021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:22.756977 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:22.890528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:22.889939 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:23.017703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:23.017667 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:23.147843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:23.147758 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:23.229786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:23.229752 4711 patch_prober.go:29] interesting pod/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 03 20:12:23.229946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:23.229801 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 03 20:12:23.848018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:23.847988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:23.950499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:23.950432 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" podUID="5ce0fb41-a836-469b-ac55-84c448c650b1" containerName="payload-processing" containerID="cri-o://4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e" gracePeriod=25 Nov 03 20:12:23.962989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:23.962951 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" podUID="59119cbf-9835-4eb1-a44d-1fb4e30e97bd" containerName="payload-pre-processing" containerID="cri-o://409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327" gracePeriod=25 Nov 03 20:12:23.979452 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:23.979424 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:24.224921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.224895 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:12:24.234146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.234119 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:12:24.234465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.234424 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:12:24.235255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.235232 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:12:24.253225 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.253209 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:12:24.320749 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.320714 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4"] Nov 03 20:12:24.321459 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.321425 4711 cpu_manager.go:414] "containerMap: removing container" podUID="59119cbf-9835-4eb1-a44d-1fb4e30e97bd" containerName="payload-pre-processing" Nov 03 20:12:24.321459 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.321459 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59119cbf-9835-4eb1-a44d-1fb4e30e97bd" containerName="payload-pre-processing" Nov 03 20:12:24.321640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.321508 4711 cpu_manager.go:414] "containerMap: removing container" podUID="5ce0fb41-a836-469b-ac55-84c448c650b1" containerName="payload-processing" Nov 03 20:12:24.321640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.321516 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce0fb41-a836-469b-ac55-84c448c650b1" containerName="payload-processing" Nov 03 20:12:24.321640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.321629 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce0fb41-a836-469b-ac55-84c448c650b1" containerName="payload-processing" Nov 03 20:12:24.321798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.321646 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="59119cbf-9835-4eb1-a44d-1fb4e30e97bd" containerName="payload-pre-processing" Nov 03 20:12:24.325321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.325302 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.330110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.330075 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4f93d1-openshift-default-dockercfg-kqdcn\"" Nov 03 20:12:24.338875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.338852 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4"] Nov 03 20:12:24.345325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.343503 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce0fb41-a836-469b-ac55-84c448c650b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce0fb41-a836-469b-ac55-84c448c650b1-plugins-config-volume\") pod \"5ce0fb41-a836-469b-ac55-84c448c650b1\" (UID: \"5ce0fb41-a836-469b-ac55-84c448c650b1\") " Nov 03 20:12:24.345325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.343541 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce0fb41-a836-469b-ac55-84c448c650b1-kube-api-access-8p5tb\" (UniqueName: \"kubernetes.io/projected/5ce0fb41-a836-469b-ac55-84c448c650b1-kube-api-access-8p5tb\") pod \"5ce0fb41-a836-469b-ac55-84c448c650b1\" (UID: \"5ce0fb41-a836-469b-ac55-84c448c650b1\") " Nov 03 20:12:24.345325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.344852 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce0fb41-a836-469b-ac55-84c448c650b1-plugins-config-volume" pod "5ce0fb41-a836-469b-ac55-84c448c650b1" (UID: "5ce0fb41-a836-469b-ac55-84c448c650b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:24.347476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.346780 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce0fb41-a836-469b-ac55-84c448c650b1-kube-api-access-8p5tb" pod "5ce0fb41-a836-469b-ac55-84c448c650b1" (UID: "5ce0fb41-a836-469b-ac55-84c448c650b1"). InnerVolumeSpecName "kube-api-access-8p5tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:24.406513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.406388 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:24.445027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.444995 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-kube-api-access-b68g8\" (UniqueName: \"kubernetes.io/projected/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-kube-api-access-b68g8\") pod \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\" (UID: \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\") " Nov 03 20:12:24.445206 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445185 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-plugins-config-volume\") pod \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\" (UID: \"59119cbf-9835-4eb1-a44d-1fb4e30e97bd\") " Nov 03 20:12:24.445386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445366 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.445540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445522 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1703639c-2138-475d-aefd-275fef6f84f6-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.445610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445560 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt26z\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-kube-api-access-pt26z\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.445610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445581 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-plugins-config-volume" pod "59119cbf-9835-4eb1-a44d-1fb4e30e97bd" (UID: "59119cbf-9835-4eb1-a44d-1fb4e30e97bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:24.445610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445592 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.445762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445737 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1703639c-2138-475d-aefd-275fef6f84f6-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.445815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445800 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.445870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445852 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.445925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445911 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.445995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.445976 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.446086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.446074 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:24.446121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.446090 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce0fb41-a836-469b-ac55-84c448c650b1-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:24.446121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.446101 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5tb\" (UniqueName: \"kubernetes.io/projected/5ce0fb41-a836-469b-ac55-84c448c650b1-kube-api-access-8p5tb\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:24.447141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.447124 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-kube-api-access-b68g8" pod "59119cbf-9835-4eb1-a44d-1fb4e30e97bd" (UID: "59119cbf-9835-4eb1-a44d-1fb4e30e97bd"). InnerVolumeSpecName "kube-api-access-b68g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:24.531126 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.531099 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh"] Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.554192 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.554250 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.554285 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.554332 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.554430 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1703639c-2138-475d-aefd-275fef6f84f6-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.555401 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.555719 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.555932 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.556130 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.554490 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt26z\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-kube-api-access-pt26z\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.557145 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.557195 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1703639c-2138-475d-aefd-275fef6f84f6-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.557238 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.558122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.557355 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b68g8\" (UniqueName: \"kubernetes.io/projected/59119cbf-9835-4eb1-a44d-1fb4e30e97bd-kube-api-access-b68g8\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:24.567514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.560192 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1703639c-2138-475d-aefd-275fef6f84f6-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.567514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.561050 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1703639c-2138-475d-aefd-275fef6f84f6-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.567514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.565294 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.568274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.568247 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:24.571463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.571424 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.578673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.578644 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt26z\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-kube-api-access-pt26z\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.646391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.646361 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:24.673136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.673096 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb"] Nov 03 20:12:24.673364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.673337 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" podUID="18286a56-9ac9-4ff2-88e0-9a2c7a91e216" containerName="maas-api" containerID="cri-o://bdb0eeb4f7d4e507cb81d97a3450acf68de5713d40ca269fd1451d20dc838435" gracePeriod=30 Nov 03 20:12:24.706931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.706888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:24.794599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.794485 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l"] Nov 03 20:12:24.822157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.822007 4711 generic.go:356] "Generic (PLEG): container finished" podID="59119cbf-9835-4eb1-a44d-1fb4e30e97bd" containerID="409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327" exitCode=0 Nov 03 20:12:24.823715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.822747 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" event={"ID":"59119cbf-9835-4eb1-a44d-1fb4e30e97bd","Type":"ContainerDied","Data":"409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327"} Nov 03 20:12:24.823715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.822819 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" event={"ID":"59119cbf-9835-4eb1-a44d-1fb4e30e97bd","Type":"ContainerDied","Data":"770c9b1c5b2d1fe93ef58e5cc0c889b2513759a54bddfdb11f2d59241460d668"} Nov 03 20:12:24.823715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.822853 4711 scope.go:122] "RemoveContainer" containerID="409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327" Nov 03 20:12:24.823715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.823034 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj" Nov 03 20:12:24.827415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.827362 4711 generic.go:356] "Generic (PLEG): container finished" podID="5ce0fb41-a836-469b-ac55-84c448c650b1" containerID="4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e" exitCode=0 Nov 03 20:12:24.827580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.827562 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" Nov 03 20:12:24.828261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.827912 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" event={"ID":"5ce0fb41-a836-469b-ac55-84c448c650b1","Type":"ContainerDied","Data":"4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e"} Nov 03 20:12:24.828261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.828016 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8" event={"ID":"5ce0fb41-a836-469b-ac55-84c448c650b1","Type":"ContainerDied","Data":"329b5d99d0e47aad3139355ca57ae9656f264e97ff30a95a5535077d9b67bb63"} Nov 03 20:12:24.831495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.831470 4711 generic.go:356] "Generic (PLEG): container finished" podID="18286a56-9ac9-4ff2-88e0-9a2c7a91e216" containerID="bdb0eeb4f7d4e507cb81d97a3450acf68de5713d40ca269fd1451d20dc838435" exitCode=0 Nov 03 20:12:24.831893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.831873 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" event={"ID":"18286a56-9ac9-4ff2-88e0-9a2c7a91e216","Type":"ContainerDied","Data":"bdb0eeb4f7d4e507cb81d97a3450acf68de5713d40ca269fd1451d20dc838435"} Nov 03 20:12:24.856084 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.855961 4711 scope.go:122] "RemoveContainer" containerID="409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327" Nov 03 20:12:24.856728 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:24.856612 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327\": container with ID starting with 409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327 not found: ID does not exist" containerID="409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327" Nov 03 20:12:24.856728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.856647 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327"} err="failed to get container status \"409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327\": rpc error: code = NotFound desc = could not find container \"409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327\": container with ID starting with 409118adb627cd09dd3df972bb2f5bc194f3a3c0cfd50199678335482eba2327 not found: ID does not exist" Nov 03 20:12:24.856728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.856669 4711 scope.go:122] "RemoveContainer" containerID="4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e" Nov 03 20:12:24.859410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.859387 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:24.868227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.868158 4711 scope.go:122] "RemoveContainer" containerID="4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e" Nov 03 20:12:24.868609 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:24.868581 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e\": container with ID starting with 4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e not found: ID does not exist" containerID="4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e" Nov 03 20:12:24.868693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.868618 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e"} err="failed to get container status \"4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e\": rpc error: code = NotFound desc = could not find container \"4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e\": container with ID starting with 4587703ae8a82552289d8f038e5480289ec16be9524e7fd589ca702a5ed27c5e not found: ID does not exist" Nov 03 20:12:24.870965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.870912 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj"] Nov 03 20:12:24.873043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.873023 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-459ca04a-77bdd67bdc-29plj"] Nov 03 20:12:24.883979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.883959 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8"] Nov 03 20:12:24.890315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.890291 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-459ca04a-59f8df6c94-szkg8"] Nov 03 20:12:24.946473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.946410 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4"] Nov 03 20:12:24.947586 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:24.947522 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1703639c_2138_475d_aefd_275fef6f84f6.slice/crio-e49d4eef9457a5c8880aa4577cad0c7ada461fd8b50f44b9a40fac23dfb78315 WatchSource:0}: Error finding container e49d4eef9457a5c8880aa4577cad0c7ada461fd8b50f44b9a40fac23dfb78315: Status 404 returned error can't find the container with id e49d4eef9457a5c8880aa4577cad0c7ada461fd8b50f44b9a40fac23dfb78315 Nov 03 20:12:24.950494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.950405 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:24.950591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.950543 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:24.950633 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.950591 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:24.988422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.988396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:24.992596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:24.992578 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:12:25.164363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.164332 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-kube-api-access-gkqwp\" (UniqueName: \"kubernetes.io/projected/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-kube-api-access-gkqwp\") pod \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " Nov 03 20:12:25.164363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.164369 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-metrics-certs\") pod \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " Nov 03 20:12:25.164611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.164431 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-openshift-service-ca\") pod \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " Nov 03 20:12:25.164611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.164587 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-maas-api-tls\") pod \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\" (UID: \"18286a56-9ac9-4ff2-88e0-9a2c7a91e216\") " Nov 03 20:12:25.164913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.164886 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-openshift-service-ca" pod "18286a56-9ac9-4ff2-88e0-9a2c7a91e216" (UID: "18286a56-9ac9-4ff2-88e0-9a2c7a91e216"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:25.166718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.166691 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-maas-api-tls" pod "18286a56-9ac9-4ff2-88e0-9a2c7a91e216" (UID: "18286a56-9ac9-4ff2-88e0-9a2c7a91e216"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:25.166816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.166736 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-metrics-certs" pod "18286a56-9ac9-4ff2-88e0-9a2c7a91e216" (UID: "18286a56-9ac9-4ff2-88e0-9a2c7a91e216"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:25.166816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.166802 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-kube-api-access-gkqwp" pod "18286a56-9ac9-4ff2-88e0-9a2c7a91e216" (UID: "18286a56-9ac9-4ff2-88e0-9a2c7a91e216"). InnerVolumeSpecName "kube-api-access-gkqwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:25.204730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.204645 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:25.265659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.265630 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:25.265659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.265655 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqwp\" (UniqueName: \"kubernetes.io/projected/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-kube-api-access-gkqwp\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:25.265659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.265665 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:25.265946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.265674 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18286a56-9ac9-4ff2-88e0-9a2c7a91e216-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:25.335905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.335868 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:25.840107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.840074 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" event={"ID":"18286a56-9ac9-4ff2-88e0-9a2c7a91e216","Type":"ContainerDied","Data":"e7e5137b391c3f7c78a50c633665fc969dcb714c75f35f13d188ebda136a85a3"} Nov 03 20:12:25.840285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.840142 4711 scope.go:122] "RemoveContainer" containerID="bdb0eeb4f7d4e507cb81d97a3450acf68de5713d40ca269fd1451d20dc838435" Nov 03 20:12:25.840285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.840079 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb" Nov 03 20:12:25.841836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.841806 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" event={"ID":"1703639c-2138-475d-aefd-275fef6f84f6","Type":"ContainerStarted","Data":"cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff"} Nov 03 20:12:25.841974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.841860 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" event={"ID":"1703639c-2138-475d-aefd-275fef6f84f6","Type":"ContainerStarted","Data":"e49d4eef9457a5c8880aa4577cad0c7ada461fd8b50f44b9a40fac23dfb78315"} Nov 03 20:12:25.868712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.868675 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" podStartSLOduration=1.868661943 podStartE2EDuration="1.868661943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:25.867187252 +0000 UTC m=+649.317293310" watchObservedRunningTime="2026-11-03 20:12:25.868661943 +0000 UTC m=+649.318767967" Nov 03 20:12:25.884410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.884384 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb"] Nov 03 20:12:25.891309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:25.891282 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e17fc195-57dfbf8484-fprkb"] Nov 03 20:12:26.190912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.190800 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:26.256720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.254281 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm"] Nov 03 20:12:26.371561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.371515 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:26.509577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.509546 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:26.646962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.646930 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:26.652348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.652324 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:26.682692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.682659 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18286a56-9ac9-4ff2-88e0-9a2c7a91e216" path="/var/lib/kubelet/pods/18286a56-9ac9-4ff2-88e0-9a2c7a91e216/volumes" Nov 03 20:12:26.683200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.683180 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59119cbf-9835-4eb1-a44d-1fb4e30e97bd" path="/var/lib/kubelet/pods/59119cbf-9835-4eb1-a44d-1fb4e30e97bd/volumes" Nov 03 20:12:26.683731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.683705 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce0fb41-a836-469b-ac55-84c448c650b1" path="/var/lib/kubelet/pods/5ce0fb41-a836-469b-ac55-84c448c650b1/volumes" Nov 03 20:12:26.847905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.847831 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:26.848766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.848745 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:26.884548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.884520 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:26.942290 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.942260 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2"] Nov 03 20:12:26.943426 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:26.943382 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" podUID="823460b4-3334-4436-9dc0-df5f5e4a323a" containerName="maas-api" containerID="cri-o://896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352" gracePeriod=30 Nov 03 20:12:27.025969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.025943 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:27.191043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.191018 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:27.261115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.261069 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:27.282113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.282076 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823460b4-3334-4436-9dc0-df5f5e4a323a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/823460b4-3334-4436-9dc0-df5f5e4a323a-openshift-service-ca\") pod \"823460b4-3334-4436-9dc0-df5f5e4a323a\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " Nov 03 20:12:27.282113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.282112 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-maas-api-tls\") pod \"823460b4-3334-4436-9dc0-df5f5e4a323a\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " Nov 03 20:12:27.282361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.282174 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823460b4-3334-4436-9dc0-df5f5e4a323a-kube-api-access-8th2x\" (UniqueName: \"kubernetes.io/projected/823460b4-3334-4436-9dc0-df5f5e4a323a-kube-api-access-8th2x\") pod \"823460b4-3334-4436-9dc0-df5f5e4a323a\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " Nov 03 20:12:27.282361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.282254 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-metrics-certs\") pod \"823460b4-3334-4436-9dc0-df5f5e4a323a\" (UID: \"823460b4-3334-4436-9dc0-df5f5e4a323a\") " Nov 03 20:12:27.282508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.282474 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823460b4-3334-4436-9dc0-df5f5e4a323a-openshift-service-ca" pod "823460b4-3334-4436-9dc0-df5f5e4a323a" (UID: "823460b4-3334-4436-9dc0-df5f5e4a323a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:27.282643 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.282626 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/823460b4-3334-4436-9dc0-df5f5e4a323a-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:27.284308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.284282 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-maas-api-tls" pod "823460b4-3334-4436-9dc0-df5f5e4a323a" (UID: "823460b4-3334-4436-9dc0-df5f5e4a323a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:27.284409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.284329 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823460b4-3334-4436-9dc0-df5f5e4a323a-kube-api-access-8th2x" pod "823460b4-3334-4436-9dc0-df5f5e4a323a" (UID: "823460b4-3334-4436-9dc0-df5f5e4a323a"). InnerVolumeSpecName "kube-api-access-8th2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:27.284409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.284394 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-metrics-certs" pod "823460b4-3334-4436-9dc0-df5f5e4a323a" (UID: "823460b4-3334-4436-9dc0-df5f5e4a323a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:27.383239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.383161 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:27.383239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.383189 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8th2x\" (UniqueName: \"kubernetes.io/projected/823460b4-3334-4436-9dc0-df5f5e4a323a-kube-api-access-8th2x\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:27.383239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.383198 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/823460b4-3334-4436-9dc0-df5f5e4a323a-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:27.388879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.388844 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:27.509172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.509141 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:27.639496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.639391 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:27.653634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.653601 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2"] Nov 03 20:12:27.653863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.653837 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" containerID="cri-o://248c09fce515b17822a68f1970a3e43159a42041bbd892b77bcfdf1cd1a25494" gracePeriod=30 Nov 03 20:12:27.772737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.772640 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:27.853802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.853768 4711 generic.go:356] "Generic (PLEG): container finished" podID="823460b4-3334-4436-9dc0-df5f5e4a323a" containerID="896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352" exitCode=0 Nov 03 20:12:27.853975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.853814 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" Nov 03 20:12:27.853975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.853845 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" event={"ID":"823460b4-3334-4436-9dc0-df5f5e4a323a","Type":"ContainerDied","Data":"896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352"} Nov 03 20:12:27.853975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.853888 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2" event={"ID":"823460b4-3334-4436-9dc0-df5f5e4a323a","Type":"ContainerDied","Data":"6d497a2d40b39800aaec3bcebc3ac931cf04ec3e1209889db7ffc70b7c808768"} Nov 03 20:12:27.853975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.853908 4711 scope.go:122] "RemoveContainer" containerID="896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352" Nov 03 20:12:27.863426 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.863408 4711 scope.go:122] "RemoveContainer" containerID="896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352" Nov 03 20:12:27.863699 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:27.863674 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352\": container with ID starting with 896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352 not found: ID does not exist" containerID="896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352" Nov 03 20:12:27.863747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.863707 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352"} err="failed to get container status \"896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352\": rpc error: code = NotFound desc = could not find container \"896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352\": container with ID starting with 896cec5732245b8b7d80479eff7c6ed96662c1cc504ea762d3cd25ad4d8d3352 not found: ID does not exist" Nov 03 20:12:27.903944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.903888 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2"] Nov 03 20:12:27.909719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.909693 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd752fd-7bb546f799-fc7p2"] Nov 03 20:12:27.954616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.954587 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc"] Nov 03 20:12:27.954814 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:27.954786 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" containerID="cri-o://252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba" gracePeriod=30 Nov 03 20:12:28.065463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.065383 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:28.078771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.078740 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2"] Nov 03 20:12:28.079472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.079435 4711 cpu_manager.go:414] "containerMap: removing container" podUID="823460b4-3334-4436-9dc0-df5f5e4a323a" containerName="maas-api" Nov 03 20:12:28.079527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.079470 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823460b4-3334-4436-9dc0-df5f5e4a323a" containerName="maas-api" Nov 03 20:12:28.079527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.079501 4711 cpu_manager.go:414] "containerMap: removing container" podUID="18286a56-9ac9-4ff2-88e0-9a2c7a91e216" containerName="maas-api" Nov 03 20:12:28.079527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.079510 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18286a56-9ac9-4ff2-88e0-9a2c7a91e216" containerName="maas-api" Nov 03 20:12:28.079648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.079637 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="823460b4-3334-4436-9dc0-df5f5e4a323a" containerName="maas-api" Nov 03 20:12:28.079702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.079653 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="18286a56-9ac9-4ff2-88e0-9a2c7a91e216" containerName="maas-api" Nov 03 20:12:28.084758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.084735 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.088910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.088885 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzjjp\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-kube-api-access-nzjjp\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.089160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.089137 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.089260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.089188 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.089328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.089250 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.089927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.089870 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.089927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.089912 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.090068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.089956 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.090068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.090034 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.090068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.090061 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.091476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.091456 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-466f83-openshift-default-dockercfg-6pzmx\"" Nov 03 20:12:28.126067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.125984 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2"] Nov 03 20:12:28.191410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191339 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzjjp\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-kube-api-access-nzjjp\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191383 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191418 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191474 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191502 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191524 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191561 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191644 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191668 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191841 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.191943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.191858 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.192087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.192034 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.192140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.192093 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.192378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.192358 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.193966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.193943 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.194116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.194095 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.207280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.207254 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzjjp\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-kube-api-access-nzjjp\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.211032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.211007 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.226315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.226291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:28.398407 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.398371 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:28.434928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.434902 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:28.525397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.525365 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:28.656056 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.656019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:28.678578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.678554 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823460b4-3334-4436-9dc0-df5f5e4a323a" path="/var/lib/kubelet/pods/823460b4-3334-4436-9dc0-df5f5e4a323a/volumes" Nov 03 20:12:28.730323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.730226 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2"] Nov 03 20:12:28.733365 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:28.733340 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4a6ea8_1083_4baf_b1da_fd59c9c96224.slice/crio-6fd0a442de6e422f502033eb3eb1c1e2158b40e741d6639d3d5f4fca051e7952 WatchSource:0}: Error finding container 6fd0a442de6e422f502033eb3eb1c1e2158b40e741d6639d3d5f4fca051e7952: Status 404 returned error can't find the container with id 6fd0a442de6e422f502033eb3eb1c1e2158b40e741d6639d3d5f4fca051e7952 Nov 03 20:12:28.735436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.735405 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:28.735551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.735505 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:28.735551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.735537 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:28.808261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.808224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:28.861471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.861412 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" event={"ID":"ff4a6ea8-1083-4baf-b1da-fd59c9c96224","Type":"ContainerStarted","Data":"1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2"} Nov 03 20:12:28.861649 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.861493 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" event={"ID":"ff4a6ea8-1083-4baf-b1da-fd59c9c96224","Type":"ContainerStarted","Data":"6fd0a442de6e422f502033eb3eb1c1e2158b40e741d6639d3d5f4fca051e7952"} Nov 03 20:12:28.959265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:28.959223 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:29.106534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.105290 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:29.257255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.257223 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:29.399621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.399525 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:29.435599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.435571 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:29.440542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.440520 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:29.465602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.465553 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" podStartSLOduration=1.465539708 podStartE2EDuration="1.465539708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:28.89082501 +0000 UTC m=+652.340931058" watchObservedRunningTime="2026-11-03 20:12:29.465539708 +0000 UTC m=+652.915645753" Nov 03 20:12:29.480029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.479986 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf"] Nov 03 20:12:29.487030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.487001 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:29.493691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.493666 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf"] Nov 03 20:12:29.493791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.493670 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-4f93d1-dockercfg-bmvch\"" Nov 03 20:12:29.493997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.493981 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-4f93d1\"" Nov 03 20:12:29.521074 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.521046 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q25h9\" (UniqueName: \"kubernetes.io/projected/7d2b99c0-6966-4d2e-aa40-19afba1be993-kube-api-access-q25h9\") pod \"payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf\" (UID: \"7d2b99c0-6966-4d2e-aa40-19afba1be993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:29.521237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.521094 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2b99c0-6966-4d2e-aa40-19afba1be993-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf\" (UID: \"7d2b99c0-6966-4d2e-aa40-19afba1be993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:29.532215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.532178 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" podUID="9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8" containerName="payload-processing" containerID="cri-o://6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60" gracePeriod=25 Nov 03 20:12:29.548504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.548479 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:29.622053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.622025 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25h9\" (UniqueName: \"kubernetes.io/projected/7d2b99c0-6966-4d2e-aa40-19afba1be993-kube-api-access-q25h9\") pod \"payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf\" (UID: \"7d2b99c0-6966-4d2e-aa40-19afba1be993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:29.622243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.622079 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2b99c0-6966-4d2e-aa40-19afba1be993-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf\" (UID: \"7d2b99c0-6966-4d2e-aa40-19afba1be993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:29.622868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.622841 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2b99c0-6966-4d2e-aa40-19afba1be993-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf\" (UID: \"7d2b99c0-6966-4d2e-aa40-19afba1be993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:29.634502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.634480 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q25h9\" (UniqueName: \"kubernetes.io/projected/7d2b99c0-6966-4d2e-aa40-19afba1be993-kube-api-access-q25h9\") pod \"payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf\" (UID: \"7d2b99c0-6966-4d2e-aa40-19afba1be993\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:29.787151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.787127 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:12:29.795365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.795325 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" podUID="dff5acad-67c8-4370-8d45-36117dd8b248" containerName="payload-pre-processing" containerID="cri-o://be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611" gracePeriod=25 Nov 03 20:12:29.802548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.802526 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:29.823411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.823383 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-plugins-config-volume\") pod \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\" (UID: \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\") " Nov 03 20:12:29.823568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.823523 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-kube-api-access-mq2hz\" (UniqueName: \"kubernetes.io/projected/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-kube-api-access-mq2hz\") pod \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\" (UID: \"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8\") " Nov 03 20:12:29.823845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.823816 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-plugins-config-volume" pod "9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8" (UID: "9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:29.823962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.823946 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:29.825595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.825570 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-kube-api-access-mq2hz" pod "9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8" (UID: "9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8"). InnerVolumeSpecName "kube-api-access-mq2hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:29.875497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.875425 4711 generic.go:356] "Generic (PLEG): container finished" podID="9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8" containerID="6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60" exitCode=0 Nov 03 20:12:29.875645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.875523 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" Nov 03 20:12:29.875645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.875588 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" event={"ID":"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8","Type":"ContainerDied","Data":"6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60"} Nov 03 20:12:29.875645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.875630 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh" event={"ID":"9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8","Type":"ContainerDied","Data":"10dc98c6a4d184a09d14876e65777da6d9dcd5b2635d03ac26a660a5ee1e4f9a"} Nov 03 20:12:29.875816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.875654 4711 scope.go:122] "RemoveContainer" containerID="6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60" Nov 03 20:12:29.876597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.876148 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:29.877668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.877651 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:29.908327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.908305 4711 scope.go:122] "RemoveContainer" containerID="6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60" Nov 03 20:12:29.908742 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:29.908721 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60\": container with ID starting with 6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60 not found: ID does not exist" containerID="6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60" Nov 03 20:12:29.908817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.908753 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60"} err="failed to get container status \"6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60\": rpc error: code = NotFound desc = could not find container \"6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60\": container with ID starting with 6e759288c83b2b16cd34cf56d6488bfb50f563095bdde99dad413d2eb3af1b60 not found: ID does not exist" Nov 03 20:12:29.923938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.923914 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9"] Nov 03 20:12:29.926462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.926415 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq2hz\" (UniqueName: \"kubernetes.io/projected/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8-kube-api-access-mq2hz\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:29.927479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.927456 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8" containerName="payload-processing" Nov 03 20:12:29.927619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.927603 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8" containerName="payload-processing" Nov 03 20:12:29.927865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.927852 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8" containerName="payload-processing" Nov 03 20:12:29.931946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.931920 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:29.941843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.941717 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:12:29.941843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.941814 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:12:29.947438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.947414 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9"] Nov 03 20:12:29.949505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.949433 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh"] Nov 03 20:12:29.952750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.952691 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e17fc195-5fdc496f48-j7dgh"] Nov 03 20:12:29.992099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.992067 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:29.994855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.994630 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqpwn\"" Nov 03 20:12:29.995435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.995318 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:12:29.995435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:29.995396 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-4f93d1-serving-cert\"" Nov 03 20:12:30.027901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.027857 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-metrics-certs\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.028142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.028117 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrzc\" (UniqueName: \"kubernetes.io/projected/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-kube-api-access-hnrzc\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.028231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.028212 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.028308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.028283 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.069625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.069598 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:12:30.082816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.082792 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf"] Nov 03 20:12:30.084310 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:30.084288 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2b99c0_6966_4d2e_aa40_19afba1be993.slice/crio-0ad7d54c87656a37daea266ab0f546cc519536ea36c704e65f3407e9b3d65703 WatchSource:0}: Error finding container 0ad7d54c87656a37daea266ab0f546cc519536ea36c704e65f3407e9b3d65703: Status 404 returned error can't find the container with id 0ad7d54c87656a37daea266ab0f546cc519536ea36c704e65f3407e9b3d65703 Nov 03 20:12:30.124371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.124345 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:30.130004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.129405 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff5acad-67c8-4370-8d45-36117dd8b248-kube-api-access-b9zxn\" (UniqueName: \"kubernetes.io/projected/dff5acad-67c8-4370-8d45-36117dd8b248-kube-api-access-b9zxn\") pod \"dff5acad-67c8-4370-8d45-36117dd8b248\" (UID: \"dff5acad-67c8-4370-8d45-36117dd8b248\") " Nov 03 20:12:30.130004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.129561 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff5acad-67c8-4370-8d45-36117dd8b248-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5acad-67c8-4370-8d45-36117dd8b248-plugins-config-volume\") pod \"dff5acad-67c8-4370-8d45-36117dd8b248\" (UID: \"dff5acad-67c8-4370-8d45-36117dd8b248\") " Nov 03 20:12:30.130004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.129741 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.130004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.129782 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.130004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.129848 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-metrics-certs\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.130345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.130010 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrzc\" (UniqueName: \"kubernetes.io/projected/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-kube-api-access-hnrzc\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.132255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.131078 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.132255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.131570 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff5acad-67c8-4370-8d45-36117dd8b248-plugins-config-volume" pod "dff5acad-67c8-4370-8d45-36117dd8b248" (UID: "dff5acad-67c8-4370-8d45-36117dd8b248"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:30.132255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.132020 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff5acad-67c8-4370-8d45-36117dd8b248-kube-api-access-b9zxn" pod "dff5acad-67c8-4370-8d45-36117dd8b248" (UID: "dff5acad-67c8-4370-8d45-36117dd8b248"). InnerVolumeSpecName "kube-api-access-b9zxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:30.132986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.132962 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.132986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.133000 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-metrics-certs\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.143013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.142985 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrzc\" (UniqueName: \"kubernetes.io/projected/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-kube-api-access-hnrzc\") pod \"maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.231359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.231329 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9zxn\" (UniqueName: \"kubernetes.io/projected/dff5acad-67c8-4370-8d45-36117dd8b248-kube-api-access-b9zxn\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:30.231490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.231365 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff5acad-67c8-4370-8d45-36117dd8b248-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:30.254955 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.254929 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:30.544841 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.544765 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9"] Nov 03 20:12:30.547132 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:30.547102 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf9dbbc_4dda_4af1_9dd6_2da60134f11d.slice/crio-98bd98c7c783c8936b952558f7a233afd04dcc5b1213a2f6ce533fbfa3bfcb57 WatchSource:0}: Error finding container 98bd98c7c783c8936b952558f7a233afd04dcc5b1213a2f6ce533fbfa3bfcb57: Status 404 returned error can't find the container with id 98bd98c7c783c8936b952558f7a233afd04dcc5b1213a2f6ce533fbfa3bfcb57 Nov 03 20:12:30.679701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.679669 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8" path="/var/lib/kubelet/pods/9bc97aa6-a8f7-43d7-a9a2-ebb256750cc8/volumes" Nov 03 20:12:30.881807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.881724 4711 generic.go:356] "Generic (PLEG): container finished" podID="dff5acad-67c8-4370-8d45-36117dd8b248" containerID="be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611" exitCode=0 Nov 03 20:12:30.881807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.881775 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" Nov 03 20:12:30.881991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.881813 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" event={"ID":"dff5acad-67c8-4370-8d45-36117dd8b248","Type":"ContainerDied","Data":"be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611"} Nov 03 20:12:30.881991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.881870 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l" event={"ID":"dff5acad-67c8-4370-8d45-36117dd8b248","Type":"ContainerDied","Data":"212a9a08cea4d88ae6a8ff40a55d04e27a557e884e8502e511e4c9040e24bb7a"} Nov 03 20:12:30.881991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.881896 4711 scope.go:122] "RemoveContainer" containerID="be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611" Nov 03 20:12:30.883764 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.883737 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" event={"ID":"7d2b99c0-6966-4d2e-aa40-19afba1be993","Type":"ContainerStarted","Data":"f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b"} Nov 03 20:12:30.883891 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.883812 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:30.883891 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.883836 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" event={"ID":"7d2b99c0-6966-4d2e-aa40-19afba1be993","Type":"ContainerStarted","Data":"0ad7d54c87656a37daea266ab0f546cc519536ea36c704e65f3407e9b3d65703"} Nov 03 20:12:30.885430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.885408 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" event={"ID":"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d","Type":"ContainerStarted","Data":"98bd98c7c783c8936b952558f7a233afd04dcc5b1213a2f6ce533fbfa3bfcb57"} Nov 03 20:12:30.891607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.891586 4711 scope.go:122] "RemoveContainer" containerID="be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611" Nov 03 20:12:30.891872 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:30.891853 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611\": container with ID starting with be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611 not found: ID does not exist" containerID="be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611" Nov 03 20:12:30.891912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.891884 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611"} err="failed to get container status \"be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611\": rpc error: code = NotFound desc = could not find container \"be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611\": container with ID starting with be61b22b2684af229e33f107217d286fd3d6f973bcb53f913dd3732feeed3611 not found: ID does not exist" Nov 03 20:12:30.902161 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.902137 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l"] Nov 03 20:12:30.911229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.911206 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9dd42e13-fc54468c-wbg5l"] Nov 03 20:12:30.929115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:30.929070 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" podStartSLOduration=1.929059562 podStartE2EDuration="1.929059562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:30.927112997 +0000 UTC m=+654.377219045" watchObservedRunningTime="2026-11-03 20:12:30.929059562 +0000 UTC m=+654.379165608" Nov 03 20:12:31.000944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.000910 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:31.234689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.234570 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:31.255541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.255495 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" podUID="5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed" containerName="payload-processing" containerID="cri-o://5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933" gracePeriod=25 Nov 03 20:12:31.458224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.458187 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:31.519472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.519434 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:12:31.649504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.649472 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-kube-api-access-lnmn7\") pod \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\" (UID: \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\") " Nov 03 20:12:31.649651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.649606 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-plugins-config-volume\") pod \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\" (UID: \"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed\") " Nov 03 20:12:31.649948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.649927 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-plugins-config-volume" pod "5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed" (UID: "5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:31.651525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.651506 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-kube-api-access-lnmn7" pod "5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed" (UID: "5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed"). InnerVolumeSpecName "kube-api-access-lnmn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:31.751003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.750923 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:31.751003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.750957 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmn7\" (UniqueName: \"kubernetes.io/projected/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed-kube-api-access-lnmn7\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:31.895174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.895141 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" event={"ID":"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d","Type":"ContainerStarted","Data":"07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08"} Nov 03 20:12:31.895326 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.895234 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:31.896512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.896485 4711 generic.go:356] "Generic (PLEG): container finished" podID="5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed" containerID="5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933" exitCode=0 Nov 03 20:12:31.896635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.896550 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" event={"ID":"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed","Type":"ContainerDied","Data":"5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933"} Nov 03 20:12:31.896635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.896597 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" event={"ID":"5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed","Type":"ContainerDied","Data":"65d50adc8971e16587488d4e6f40c743db69824ba26e0b74806b72b2842e1c98"} Nov 03 20:12:31.896635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.896622 4711 scope.go:122] "RemoveContainer" containerID="5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933" Nov 03 20:12:31.896784 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.896664 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm" Nov 03 20:12:31.905481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.905432 4711 scope.go:122] "RemoveContainer" containerID="5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933" Nov 03 20:12:31.905782 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:31.905756 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933\": container with ID starting with 5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933 not found: ID does not exist" containerID="5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933" Nov 03 20:12:31.905863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.905793 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933"} err="failed to get container status \"5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933\": rpc error: code = NotFound desc = could not find container \"5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933\": container with ID starting with 5f4e38c61c2267819c2c733e6fc11c0f25732f6f7fdd6609e829f9c70123f933 not found: ID does not exist" Nov 03 20:12:31.917787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.917748 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" podStartSLOduration=2.456848746 podStartE2EDuration="2.917737207s" totalImagesPullingTime="460.888461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:31.91642316 +0000 UTC m=+655.366529207" watchObservedRunningTime="2026-11-03 20:12:31.917737207 +0000 UTC m=+655.367843251" Nov 03 20:12:31.933147 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.933127 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm"] Nov 03 20:12:31.941749 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.941723 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd752fd-77b8585f66-jxtsm"] Nov 03 20:12:31.949759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:31.949730 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:32.085070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.085034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:32.512063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.512032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:32.647649 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.647615 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:32.679550 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.679526 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed" path="/var/lib/kubelet/pods/5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed/volumes" Nov 03 20:12:32.679930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.679915 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff5acad-67c8-4370-8d45-36117dd8b248" path="/var/lib/kubelet/pods/dff5acad-67c8-4370-8d45-36117dd8b248/volumes" Nov 03 20:12:32.774001 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.773971 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:32.911582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.911550 4711 generic.go:356] "Generic (PLEG): container finished" podID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerID="248c09fce515b17822a68f1970a3e43159a42041bbd892b77bcfdf1cd1a25494" exitCode=0 Nov 03 20:12:32.911762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.911685 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" event={"ID":"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8","Type":"ContainerDied","Data":"248c09fce515b17822a68f1970a3e43159a42041bbd892b77bcfdf1cd1a25494"} Nov 03 20:12:32.911835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.911769 4711 scope.go:122] "RemoveContainer" containerID="42c5b20882c6ed087ed5954b743a149127d0aa210b98ce8aca563180104fb055" Nov 03 20:12:32.935819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.935790 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:32.952381 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:32.952361 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c5b20882c6ed087ed5954b743a149127d0aa210b98ce8aca563180104fb055\": container with ID starting with 42c5b20882c6ed087ed5954b743a149127d0aa210b98ce8aca563180104fb055 not found: ID does not exist" containerID="42c5b20882c6ed087ed5954b743a149127d0aa210b98ce8aca563180104fb055" Nov 03 20:12:32.955974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:32.955955 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:12:33.077210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077182 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-certs\") pod \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " Nov 03 20:12:33.077339 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077272 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-credential-socket\") pod \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " Nov 03 20:12:33.077404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077353 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-envoy\") pod \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " Nov 03 20:12:33.077404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077388 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istiod-ca-cert\") pod \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " Nov 03 20:12:33.077516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077425 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-podinfo\") pod \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " Nov 03 20:12:33.077516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077465 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-certs" pod "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" (UID: "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.077516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077495 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-token\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-token\") pod \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " Nov 03 20:12:33.077677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077538 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-data\") pod \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " Nov 03 20:12:33.077677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077628 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-socket\") pod \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " Nov 03 20:12:33.077677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077645 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-credential-socket" pod "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" (UID: "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.077824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077689 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-kube-api-access-7c4rn\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-kube-api-access-7c4rn\") pod \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\" (UID: \"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8\") " Nov 03 20:12:33.077964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.077939 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istiod-ca-cert" pod "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" (UID: "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:33.078118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.078095 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.078184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.078122 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.078184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.078138 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.078576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.078546 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.078988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.078965 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-data" pod "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" (UID: "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.079963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.079926 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-envoy" pod "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" (UID: "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.080090 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.080077 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-socket" pod "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" (UID: "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.081054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.081030 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-token" pod "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" (UID: "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:33.081179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.081157 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-podinfo" pod "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" (UID: "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:12:33.081573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.081553 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-kube-api-access-7c4rn" pod "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" (UID: "ba71fb4e-7cb0-4d13-a749-3548bed0e4d8"). InnerVolumeSpecName "kube-api-access-7c4rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:33.206130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.206012 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.206130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.206043 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.206130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.206058 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.206130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.206072 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.206130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.206086 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.206130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.206101 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4rn\" (UniqueName: \"kubernetes.io/projected/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8-kube-api-access-7c4rn\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.217178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.217047 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.270870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.270846 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:12:33.407928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.407851 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/304d6315-c67b-457d-ba22-f746bf08a957-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304d6315-c67b-457d-ba22-f746bf08a957-istio-podinfo\") pod \"304d6315-c67b-457d-ba22-f746bf08a957\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " Nov 03 20:12:33.407928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.407894 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-envoy\") pod \"304d6315-c67b-457d-ba22-f746bf08a957\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " Nov 03 20:12:33.407928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.407926 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-data\") pod \"304d6315-c67b-457d-ba22-f746bf08a957\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " Nov 03 20:12:33.408200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.407969 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-istio-token\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-istio-token\") pod \"304d6315-c67b-457d-ba22-f746bf08a957\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " Nov 03 20:12:33.408200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408030 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-kube-api-access-5x5kl\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-kube-api-access-5x5kl\") pod \"304d6315-c67b-457d-ba22-f746bf08a957\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " Nov 03 20:12:33.408200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408070 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-socket\") pod \"304d6315-c67b-457d-ba22-f746bf08a957\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " Nov 03 20:12:33.408200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408148 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-certs\") pod \"304d6315-c67b-457d-ba22-f746bf08a957\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " Nov 03 20:12:33.408200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408196 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-credential-socket\") pod \"304d6315-c67b-457d-ba22-f746bf08a957\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " Nov 03 20:12:33.408471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408269 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304d6315-c67b-457d-ba22-f746bf08a957-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304d6315-c67b-457d-ba22-f746bf08a957-istiod-ca-cert\") pod \"304d6315-c67b-457d-ba22-f746bf08a957\" (UID: \"304d6315-c67b-457d-ba22-f746bf08a957\") " Nov 03 20:12:33.408471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408267 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-data" pod "304d6315-c67b-457d-ba22-f746bf08a957" (UID: "304d6315-c67b-457d-ba22-f746bf08a957"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.408471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408351 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-socket" pod "304d6315-c67b-457d-ba22-f746bf08a957" (UID: "304d6315-c67b-457d-ba22-f746bf08a957"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.408636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408503 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-credential-socket" pod "304d6315-c67b-457d-ba22-f746bf08a957" (UID: "304d6315-c67b-457d-ba22-f746bf08a957"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.408636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408519 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-certs" pod "304d6315-c67b-457d-ba22-f746bf08a957" (UID: "304d6315-c67b-457d-ba22-f746bf08a957"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.408825 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408795 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.408825 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408821 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.408924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408801 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304d6315-c67b-457d-ba22-f746bf08a957-istiod-ca-cert" pod "304d6315-c67b-457d-ba22-f746bf08a957" (UID: "304d6315-c67b-457d-ba22-f746bf08a957"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:33.408924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408837 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.408924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.408890 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.410228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.410200 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-envoy" pod "304d6315-c67b-457d-ba22-f746bf08a957" (UID: "304d6315-c67b-457d-ba22-f746bf08a957"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:12:33.410499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.410475 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-istio-token" pod "304d6315-c67b-457d-ba22-f746bf08a957" (UID: "304d6315-c67b-457d-ba22-f746bf08a957"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:33.410591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.410538 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/304d6315-c67b-457d-ba22-f746bf08a957-istio-podinfo" pod "304d6315-c67b-457d-ba22-f746bf08a957" (UID: "304d6315-c67b-457d-ba22-f746bf08a957"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:12:33.410658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.410635 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-kube-api-access-5x5kl" pod "304d6315-c67b-457d-ba22-f746bf08a957" (UID: "304d6315-c67b-457d-ba22-f746bf08a957"). InnerVolumeSpecName "kube-api-access-5x5kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:33.509936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.509906 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/304d6315-c67b-457d-ba22-f746bf08a957-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.509936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.509932 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/304d6315-c67b-457d-ba22-f746bf08a957-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.509936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.509941 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/304d6315-c67b-457d-ba22-f746bf08a957-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.510174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.509950 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.510174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.509959 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x5kl\" (UniqueName: \"kubernetes.io/projected/304d6315-c67b-457d-ba22-f746bf08a957-kube-api-access-5x5kl\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:33.525052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525020 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz"] Nov 03 20:12:33.525734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525710 4711 cpu_manager.go:414] "containerMap: removing container" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" Nov 03 20:12:33.525734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525734 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525748 4711 cpu_manager.go:414] "containerMap: removing container" podUID="dff5acad-67c8-4370-8d45-36117dd8b248" containerName="payload-pre-processing" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525756 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff5acad-67c8-4370-8d45-36117dd8b248" containerName="payload-pre-processing" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525772 4711 cpu_manager.go:414] "containerMap: removing container" podUID="5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed" containerName="payload-processing" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525779 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed" containerName="payload-processing" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525811 4711 cpu_manager.go:414] "containerMap: removing container" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525819 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525837 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525844 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525871 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" Nov 03 20:12:33.525901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525878 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" Nov 03 20:12:33.526356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.525997 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" Nov 03 20:12:33.526356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.526012 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" Nov 03 20:12:33.526356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.526020 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff5acad-67c8-4370-8d45-36117dd8b248" containerName="payload-pre-processing" Nov 03 20:12:33.526356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.526029 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e86b7ec-ee6a-44ea-a6e7-9acb448e03ed" containerName="payload-processing" Nov 03 20:12:33.526356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.526040 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="304d6315-c67b-457d-ba22-f746bf08a957" containerName="istio-proxy" Nov 03 20:12:33.526356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.526050 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" containerName="istio-proxy" Nov 03 20:12:33.530141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.530118 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.540544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.539848 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-466f83-serving-cert\"" Nov 03 20:12:33.540544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.540298 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz"] Nov 03 20:12:33.610206 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.610168 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.610511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.610489 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-metrics-certs\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.610586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.610522 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.610586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.610550 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.610664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.610648 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbrw\" (UniqueName: \"kubernetes.io/projected/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-kube-api-access-qxbrw\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.711687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.711658 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-metrics-certs\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.711848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.711692 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.711848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.711715 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.711848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.711745 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbrw\" (UniqueName: \"kubernetes.io/projected/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-kube-api-access-qxbrw\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.712319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.712294 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.714114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.714092 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-metrics-certs\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.714214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.714155 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.723905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.723884 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbrw\" (UniqueName: \"kubernetes.io/projected/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-kube-api-access-qxbrw\") pod \"maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.760340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.760310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.846565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.846540 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:33.893412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.893375 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:33.922380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.922354 4711 generic.go:356] "Generic (PLEG): container finished" podID="304d6315-c67b-457d-ba22-f746bf08a957" containerID="252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba" exitCode=0 Nov 03 20:12:33.922514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.922435 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" Nov 03 20:12:33.922514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.922474 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" event={"ID":"304d6315-c67b-457d-ba22-f746bf08a957","Type":"ContainerDied","Data":"252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba"} Nov 03 20:12:33.922624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.922528 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc" event={"ID":"304d6315-c67b-457d-ba22-f746bf08a957","Type":"ContainerDied","Data":"dc2b5366aac79ba39ab2b557e5ca555fd329e889c406a50bff8f01229409b738"} Nov 03 20:12:33.922624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.922551 4711 scope.go:122] "RemoveContainer" containerID="252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba" Nov 03 20:12:33.923925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.923786 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" event={"ID":"ba71fb4e-7cb0-4d13-a749-3548bed0e4d8","Type":"ContainerDied","Data":"cc232d798b20ad95d340e8814b655ac624f8785ee057ae15ad4e34a6f0808cda"} Nov 03 20:12:33.923925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.923808 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2" Nov 03 20:12:33.936287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.935570 4711 scope.go:122] "RemoveContainer" containerID="11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1" Nov 03 20:12:33.945957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.945930 4711 scope.go:122] "RemoveContainer" containerID="252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba" Nov 03 20:12:33.946248 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:33.946221 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba\": container with ID starting with 252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba not found: ID does not exist" containerID="252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba" Nov 03 20:12:33.946360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.946253 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba"} err="failed to get container status \"252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba\": rpc error: code = NotFound desc = could not find container \"252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba\": container with ID starting with 252d444bea33164d3fd5ad6ef17005ea6e17e9859728962ccef3b07daf160cba not found: ID does not exist" Nov 03 20:12:33.946360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.946269 4711 scope.go:122] "RemoveContainer" containerID="11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1" Nov 03 20:12:33.946558 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:33.946536 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1\": container with ID starting with 11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1 not found: ID does not exist" containerID="11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1" Nov 03 20:12:33.946619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.946566 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1"} err="failed to get container status \"11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1\": rpc error: code = NotFound desc = could not find container \"11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1\": container with ID starting with 11fb1a3efb96a64b42c411c0e9a1f57e8f0c2b048adfa54bea0357af7b713ca1 not found: ID does not exist" Nov 03 20:12:33.946619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.946586 4711 scope.go:122] "RemoveContainer" containerID="248c09fce515b17822a68f1970a3e43159a42041bbd892b77bcfdf1cd1a25494" Nov 03 20:12:33.957029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.957006 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc"] Nov 03 20:12:33.960525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.960503 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4bd752fd-openshift-default-6896d84957-rwdwc"] Nov 03 20:12:33.969318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.969295 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2"] Nov 03 20:12:33.973546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:33.973524 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e17fc195-openshift-default-67d5d7cc59-gjtg2"] Nov 03 20:12:34.158389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.158362 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz"] Nov 03 20:12:34.160396 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:34.160373 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4313d2_0e97_461e_9bd4_b698ce1daa2b.slice/crio-2941c20b0222ade22ea817a98b42c2bf7688c2ae08b03ffaad0e6969096e0ca2 WatchSource:0}: Error finding container 2941c20b0222ade22ea817a98b42c2bf7688c2ae08b03ffaad0e6969096e0ca2: Status 404 returned error can't find the container with id 2941c20b0222ade22ea817a98b42c2bf7688c2ae08b03ffaad0e6969096e0ca2 Nov 03 20:12:34.284207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.284163 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc"] Nov 03 20:12:34.289729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.289699 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:34.290078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.290059 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:34.313368 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.313343 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f3631f\"" Nov 03 20:12:34.314998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.314974 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc"] Nov 03 20:12:34.418410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.418372 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-929tm\" (UniqueName: \"kubernetes.io/projected/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-kube-api-access-929tm\") pod \"payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc\" (UID: \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:34.418589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.418428 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc\" (UID: \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:34.443089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.443052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:34.519712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.519637 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-929tm\" (UniqueName: \"kubernetes.io/projected/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-kube-api-access-929tm\") pod \"payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc\" (UID: \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:34.519880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.519759 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc\" (UID: \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:34.520390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.520368 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc\" (UID: \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:34.534390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.534324 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-929tm\" (UniqueName: \"kubernetes.io/projected/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-kube-api-access-929tm\") pod \"payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc\" (UID: \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:34.602193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.602169 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:34.686892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.685837 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304d6315-c67b-457d-ba22-f746bf08a957" path="/var/lib/kubelet/pods/304d6315-c67b-457d-ba22-f746bf08a957/volumes" Nov 03 20:12:34.686892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.686532 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba71fb4e-7cb0-4d13-a749-3548bed0e4d8" path="/var/lib/kubelet/pods/ba71fb4e-7cb0-4d13-a749-3548bed0e4d8/volumes" Nov 03 20:12:34.791277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.791175 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:34.890600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.886155 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc"] Nov 03 20:12:34.896325 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:34.896296 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3338f3e3_2bda_4b6b_8fa5_ae3e4fb82d9b.slice/crio-252232ae6ad5a402dfe305b98d69aaa917a853908dbb104ee23c2f35b9f65f6f WatchSource:0}: Error finding container 252232ae6ad5a402dfe305b98d69aaa917a853908dbb104ee23c2f35b9f65f6f: Status 404 returned error can't find the container with id 252232ae6ad5a402dfe305b98d69aaa917a853908dbb104ee23c2f35b9f65f6f Nov 03 20:12:34.931248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.931222 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" event={"ID":"8c4313d2-0e97-461e-9bd4-b698ce1daa2b","Type":"ContainerStarted","Data":"738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de"} Nov 03 20:12:34.931363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.931280 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" event={"ID":"8c4313d2-0e97-461e-9bd4-b698ce1daa2b","Type":"ContainerStarted","Data":"2941c20b0222ade22ea817a98b42c2bf7688c2ae08b03ffaad0e6969096e0ca2"} Nov 03 20:12:34.931418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.931371 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:34.933431 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.933411 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" event={"ID":"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b","Type":"ContainerStarted","Data":"252232ae6ad5a402dfe305b98d69aaa917a853908dbb104ee23c2f35b9f65f6f"} Nov 03 20:12:34.953496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:34.953430 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" podStartSLOduration=1.566789413 podStartE2EDuration="1.953415911s" totalImagesPullingTime="386.626498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:34.951419106 +0000 UTC m=+658.401525152" watchObservedRunningTime="2026-11-03 20:12:34.953415911 +0000 UTC m=+658.403521956" Nov 03 20:12:35.253334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.253245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:35.297867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.297832 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc"] Nov 03 20:12:35.301611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.301589 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:35.309052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.309028 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-085566-dockercfg-kp5zf\"" Nov 03 20:12:35.309052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.309042 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-085566\"" Nov 03 20:12:35.310068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.308552 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc"] Nov 03 20:12:35.319478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.319440 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt"] Nov 03 20:12:35.323297 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.323278 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:35.337644 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.337616 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt"] Nov 03 20:12:35.392047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.392009 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:35.429837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.429800 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qbgj\" (UniqueName: \"kubernetes.io/projected/99d18155-451a-494b-ad61-9d56bf91ad32-kube-api-access-9qbgj\") pod \"payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc\" (UID: \"99d18155-451a-494b-ad61-9d56bf91ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:35.430023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.429880 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg7fc\" (UniqueName: \"kubernetes.io/projected/2acc379e-4a07-4f2b-ac95-1f915baa79a9-kube-api-access-zg7fc\") pod \"payload-processing-e2e-worker-w2-085566-64777db978-9bjqt\" (UID: \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:35.430023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.429907 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d18155-451a-494b-ad61-9d56bf91ad32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc\" (UID: \"99d18155-451a-494b-ad61-9d56bf91ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:35.430023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.430019 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2acc379e-4a07-4f2b-ac95-1f915baa79a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-085566-64777db978-9bjqt\" (UID: \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:35.530873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.530841 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qbgj\" (UniqueName: \"kubernetes.io/projected/99d18155-451a-494b-ad61-9d56bf91ad32-kube-api-access-9qbgj\") pod \"payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc\" (UID: \"99d18155-451a-494b-ad61-9d56bf91ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:35.531026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.530901 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg7fc\" (UniqueName: \"kubernetes.io/projected/2acc379e-4a07-4f2b-ac95-1f915baa79a9-kube-api-access-zg7fc\") pod \"payload-processing-e2e-worker-w2-085566-64777db978-9bjqt\" (UID: \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:35.531026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.530935 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d18155-451a-494b-ad61-9d56bf91ad32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc\" (UID: \"99d18155-451a-494b-ad61-9d56bf91ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:35.531026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.531007 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2acc379e-4a07-4f2b-ac95-1f915baa79a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-085566-64777db978-9bjqt\" (UID: \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:35.531788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.531760 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2acc379e-4a07-4f2b-ac95-1f915baa79a9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-085566-64777db978-9bjqt\" (UID: \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:35.531905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.531842 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d18155-451a-494b-ad61-9d56bf91ad32-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc\" (UID: \"99d18155-451a-494b-ad61-9d56bf91ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:35.548896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.548867 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg7fc\" (UniqueName: \"kubernetes.io/projected/2acc379e-4a07-4f2b-ac95-1f915baa79a9-kube-api-access-zg7fc\") pod \"payload-processing-e2e-worker-w2-085566-64777db978-9bjqt\" (UID: \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:35.549014 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.548920 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qbgj\" (UniqueName: \"kubernetes.io/projected/99d18155-451a-494b-ad61-9d56bf91ad32-kube-api-access-9qbgj\") pod \"payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc\" (UID: \"99d18155-451a-494b-ad61-9d56bf91ad32\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:35.620605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.619614 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:35.621762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.621732 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv"] Nov 03 20:12:35.634606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.634583 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.638361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.638339 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:35.643029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.643007 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv"] Nov 03 20:12:35.650131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.649777 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f3631f-serving-cert\"" Nov 03 20:12:35.710666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.710186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:35.733492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.733424 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.733893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.733868 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.734124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.734052 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-metrics-certs\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.734188 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.734144 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwgcg\" (UniqueName: \"kubernetes.io/projected/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-kube-api-access-dwgcg\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.835864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.835765 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-metrics-certs\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.835864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.835825 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwgcg\" (UniqueName: \"kubernetes.io/projected/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-kube-api-access-dwgcg\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.836105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.835971 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.836105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.836064 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.838686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.838634 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.839337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.839293 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.840055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.840016 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-metrics-certs\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.845231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.845204 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:35.848808 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.848788 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwgcg\" (UniqueName: \"kubernetes.io/projected/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-kube-api-access-dwgcg\") pod \"maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.926751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.926724 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc"] Nov 03 20:12:35.928366 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:35.928336 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d18155_451a_494b_ad61_9d56bf91ad32.slice/crio-07df225904e7c693268c946bc4346bf0aa310acbe2d0f1b396f34a9d7ba855ee WatchSource:0}: Error finding container 07df225904e7c693268c946bc4346bf0aa310acbe2d0f1b396f34a9d7ba855ee: Status 404 returned error can't find the container with id 07df225904e7c693268c946bc4346bf0aa310acbe2d0f1b396f34a9d7ba855ee Nov 03 20:12:35.945945 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.945915 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" event={"ID":"99d18155-451a-494b-ad61-9d56bf91ad32","Type":"ContainerStarted","Data":"07df225904e7c693268c946bc4346bf0aa310acbe2d0f1b396f34a9d7ba855ee"} Nov 03 20:12:35.947789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.947760 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" event={"ID":"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b","Type":"ContainerStarted","Data":"f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891"} Nov 03 20:12:35.948111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.947996 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:35.956922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.956896 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:35.960747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.960728 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt"] Nov 03 20:12:35.994352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:35.994289 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" podStartSLOduration=1.994267824 podStartE2EDuration="1.994267824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:35.993079905 +0000 UTC m=+659.443185952" watchObservedRunningTime="2026-11-03 20:12:35.994267824 +0000 UTC m=+659.444373869" Nov 03 20:12:36.279482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.279295 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv"] Nov 03 20:12:36.282116 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:36.282091 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9bf0e3_5e74_45af_90d6_0f0997cfcd08.slice/crio-470d4a5af8fd3f0e1764807feaf3cf6ef11963c9f4ff99887a17be350255bc98 WatchSource:0}: Error finding container 470d4a5af8fd3f0e1764807feaf3cf6ef11963c9f4ff99887a17be350255bc98: Status 404 returned error can't find the container with id 470d4a5af8fd3f0e1764807feaf3cf6ef11963c9f4ff99887a17be350255bc98 Nov 03 20:12:36.524490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.524439 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:36.651027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.650952 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:36.954725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.954632 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" event={"ID":"99d18155-451a-494b-ad61-9d56bf91ad32","Type":"ContainerStarted","Data":"9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0"} Nov 03 20:12:36.954886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.954738 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:36.956424 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.956393 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" event={"ID":"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08","Type":"ContainerStarted","Data":"8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9"} Nov 03 20:12:36.956558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.956454 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" event={"ID":"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08","Type":"ContainerStarted","Data":"470d4a5af8fd3f0e1764807feaf3cf6ef11963c9f4ff99887a17be350255bc98"} Nov 03 20:12:36.958009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.957981 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" event={"ID":"2acc379e-4a07-4f2b-ac95-1f915baa79a9","Type":"ContainerStarted","Data":"390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb"} Nov 03 20:12:36.958104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.958031 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" event={"ID":"2acc379e-4a07-4f2b-ac95-1f915baa79a9","Type":"ContainerStarted","Data":"7397314f041eb2e4e0e3c3161fa0429ed8ec13f6364735e458d22afa32784339"} Nov 03 20:12:36.958190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.958165 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:36.987468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:36.984933 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" podStartSLOduration=1.984915626 podStartE2EDuration="1.984915626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:36.977809126 +0000 UTC m=+660.427915178" watchObservedRunningTime="2026-11-03 20:12:36.984915626 +0000 UTC m=+660.435021670" Nov 03 20:12:37.010871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:37.010822 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" podStartSLOduration=2.010809454 podStartE2EDuration="2.010809454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:37.010628946 +0000 UTC m=+660.460734993" watchObservedRunningTime="2026-11-03 20:12:37.010809454 +0000 UTC m=+660.460915498" Nov 03 20:12:37.033317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:37.033273 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" podStartSLOduration=1.624227654 podStartE2EDuration="2.033260724s" totalImagesPullingTime="409.03307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:37.032900535 +0000 UTC m=+660.483006585" watchObservedRunningTime="2026-11-03 20:12:37.033260724 +0000 UTC m=+660.483366770" Nov 03 20:12:37.593017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:37.592982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:37.731393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:37.731361 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:37.921007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:37.920933 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 20:12:37.963619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:37.963589 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:38.731904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:38.731864 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:39.114016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.113982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:39.225982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.225946 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m"] Nov 03 20:12:39.231404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.231383 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:39.242700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.242676 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m"] Nov 03 20:12:39.263555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.263524 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:39.298102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.298070 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-b13a14\"" Nov 03 20:12:39.298490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.298467 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-b13a14-dockercfg-qlnfs\"" Nov 03 20:12:39.405169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.405070 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7rzj\" (UniqueName: \"kubernetes.io/projected/c64d74d3-b7bd-491e-84db-07c449ae1946-kube-api-access-j7rzj\") pod \"payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m\" (UID: \"c64d74d3-b7bd-491e-84db-07c449ae1946\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:39.405321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.405248 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64d74d3-b7bd-491e-84db-07c449ae1946-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m\" (UID: \"c64d74d3-b7bd-491e-84db-07c449ae1946\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:39.506013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.505974 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64d74d3-b7bd-491e-84db-07c449ae1946-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m\" (UID: \"c64d74d3-b7bd-491e-84db-07c449ae1946\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:39.506258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.506052 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7rzj\" (UniqueName: \"kubernetes.io/projected/c64d74d3-b7bd-491e-84db-07c449ae1946-kube-api-access-j7rzj\") pod \"payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m\" (UID: \"c64d74d3-b7bd-491e-84db-07c449ae1946\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:39.507225 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.507195 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64d74d3-b7bd-491e-84db-07c449ae1946-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m\" (UID: \"c64d74d3-b7bd-491e-84db-07c449ae1946\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:39.518691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.518669 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7rzj\" (UniqueName: \"kubernetes.io/projected/c64d74d3-b7bd-491e-84db-07c449ae1946-kube-api-access-j7rzj\") pod \"payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m\" (UID: \"c64d74d3-b7bd-491e-84db-07c449ae1946\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:39.546867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.546844 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:39.854470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.853732 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m"] Nov 03 20:12:39.862104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:39.862031 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64d74d3_b7bd_491e_84db_07c449ae1946.slice/crio-58d3bd271c9cc9d7cd6d63eeac8854f061c1a798bd5e7405159f67e2f4602ffb WatchSource:0}: Error finding container 58d3bd271c9cc9d7cd6d63eeac8854f061c1a798bd5e7405159f67e2f4602ffb: Status 404 returned error can't find the container with id 58d3bd271c9cc9d7cd6d63eeac8854f061c1a798bd5e7405159f67e2f4602ffb Nov 03 20:12:39.920978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:39.920946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:40.014755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:40.014725 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" event={"ID":"c64d74d3-b7bd-491e-84db-07c449ae1946","Type":"ContainerStarted","Data":"58d3bd271c9cc9d7cd6d63eeac8854f061c1a798bd5e7405159f67e2f4602ffb"} Nov 03 20:12:40.892018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:40.891978 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:40.953298 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:40.953267 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 20:12:41.025475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:41.025409 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" event={"ID":"c64d74d3-b7bd-491e-84db-07c449ae1946","Type":"ContainerStarted","Data":"bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa"} Nov 03 20:12:41.025632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:41.025506 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:41.046175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:41.046116 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" podStartSLOduration=2.046099922 podStartE2EDuration="2.046099922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:41.044408231 +0000 UTC m=+664.494514279" watchObservedRunningTime="2026-11-03 20:12:41.046099922 +0000 UTC m=+664.496205989" Nov 03 20:12:41.751783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:41.751748 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:41.887471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:41.885963 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:41.900194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:41.900166 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 20:12:42.013668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:42.013636 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:42.822057 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:42.822025 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:42.890478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:42.889570 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:12:42.890478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:42.889640 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:12:43.197397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:43.197319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:43.810336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:43.810291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:43.937721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:43.937681 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:43.997371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:43.997340 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 20:12:44.092051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:44.091916 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:44.959652 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:44.959618 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:45.051157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:45.051129 4711 generic.go:356] "Generic (PLEG): container finished" podID="1703639c-2138-475d-aefd-275fef6f84f6" containerID="cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff" exitCode=0 Nov 03 20:12:45.051329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:45.051215 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" event={"ID":"1703639c-2138-475d-aefd-275fef6f84f6","Type":"ContainerDied","Data":"cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff"} Nov 03 20:12:45.051644 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:45.051626 4711 scope.go:122] "RemoveContainer" containerID="cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff" Nov 03 20:12:45.052964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:45.052937 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:45.053070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:45.053019 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:45.053070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:45.053064 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:45.951613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:45.951576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:46.059965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:46.059935 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" event={"ID":"1703639c-2138-475d-aefd-275fef6f84f6","Type":"ContainerStarted","Data":"8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23"} Nov 03 20:12:46.129680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:46.129592 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:46.646648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:46.646613 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:46.646916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:46.646841 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:12:46.646916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:46.646896 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:12:46.915991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:46.915893 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:46.961426 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:46.961399 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 20:12:47.531811 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:12:47.531777 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:12:47.647140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:47.647105 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:12:47.647309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:47.647165 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:12:47.901487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:47.901427 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:47.993476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:47.992642 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 20:12:47.994775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:47.994107 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 20:12:48.647735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:48.647704 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:12:48.648099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:48.647768 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:12:48.956331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:48.956247 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:49.647650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:49.647570 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:12:49.647650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:49.647626 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:12:49.951002 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:49.950966 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:50.119205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:50.119171 4711 generic.go:356] "Generic (PLEG): container finished" podID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerID="1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2" exitCode=0 Nov 03 20:12:50.119388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:50.119244 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" event={"ID":"ff4a6ea8-1083-4baf-b1da-fd59c9c96224","Type":"ContainerDied","Data":"1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2"} Nov 03 20:12:50.119708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:50.119689 4711 scope.go:122] "RemoveContainer" containerID="1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2" Nov 03 20:12:50.121015 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:50.120985 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:50.121076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:50.121056 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:50.121112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:50.121088 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:50.647617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:50.647585 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 03 20:12:50.647822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:50.647649 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 03 20:12:51.043829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:51.043794 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:51.127714 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:51.127678 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" event={"ID":"ff4a6ea8-1083-4baf-b1da-fd59c9c96224","Type":"ContainerStarted","Data":"372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a"} Nov 03 20:12:51.331285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:51.331176 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:51.435198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:51.435147 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:51.435399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:51.435359 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:12:51.435523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:51.435406 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:12:51.654331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:51.654250 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:51.654655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:51.654634 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:51.655397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:51.655376 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:12:52.031105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:52.031072 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:52.034533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:52.034511 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 20:12:52.191359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:52.191321 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:52.322796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:52.322709 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:52.435937 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:52.435897 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:12:52.436137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:52.435987 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:12:52.461576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:52.461544 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:52.620966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:52.620886 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:52.888171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:52.888031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:53.247610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.247519 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:53.384468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.378932 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:53.436162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.436130 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:12:53.436316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.436208 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:12:53.522709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.522678 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:53.744100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.744063 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:53.802805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.802725 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km"] Nov 03 20:12:53.805901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.805880 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:53.821506 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.821478 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km"] Nov 03 20:12:53.833820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.833794 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"openshift-service-ca.crt\"" Nov 03 20:12:53.833960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.833807 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"kube-root-ca.crt\"" Nov 03 20:12:53.845432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.845413 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"default-dockercfg-4lmjg\"" Nov 03 20:12:53.845545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.845438 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"simulator-w4-466f83-kserve-self-signed-certs\"" Nov 03 20:12:53.930771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.930736 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-home\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:53.930918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.930801 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-dshm\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:53.930978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.930916 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9f9\" (UniqueName: \"kubernetes.io/projected/4cf4fae2-dcad-47e2-ae65-d698bb535dab-kube-api-access-sx9f9\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:53.930978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.930956 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-model-cache\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:53.931092 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.931079 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tmp-dir\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:53.931134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.931105 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tls-certs\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:53.961395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:53.961358 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:54.032366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.032325 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tmp-dir\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.033278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.033256 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tls-certs\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.033933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.033912 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-home\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.034434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.034416 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-dshm\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.034985 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.034964 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9f9\" (UniqueName: \"kubernetes.io/projected/4cf4fae2-dcad-47e2-ae65-d698bb535dab-kube-api-access-sx9f9\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.038259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.035542 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-model-cache\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.038259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.034304 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-home\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.038259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.035960 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-model-cache\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.038259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.036293 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tls-certs\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.038259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.036926 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-dshm\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.038259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.033173 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tmp-dir\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.051644 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.051616 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9f9\" (UniqueName: \"kubernetes.io/projected/4cf4fae2-dcad-47e2-ae65-d698bb535dab-kube-api-access-sx9f9\") pod \"simulator-w4-466f83-kserve-6498bf78fd-ph4km\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.119398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.119309 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:54.434167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.434142 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km"] Nov 03 20:12:54.435769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.435741 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:12:54.435873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.435803 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:12:54.435873 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:12:54.435821 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cf4fae2_dcad_47e2_ae65_d698bb535dab.slice/crio-9626eb4e6ffb952392289b8d142d17381635fc12dcfa070236db8b3f2066bfdc WatchSource:0}: Error finding container 9626eb4e6ffb952392289b8d142d17381635fc12dcfa070236db8b3f2066bfdc: Status 404 returned error can't find the container with id 9626eb4e6ffb952392289b8d142d17381635fc12dcfa070236db8b3f2066bfdc Nov 03 20:12:54.791881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:54.791840 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:55.110063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:55.109966 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:55.153500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:55.153467 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" event={"ID":"4cf4fae2-dcad-47e2-ae65-d698bb535dab","Type":"ContainerStarted","Data":"3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca"} Nov 03 20:12:55.153666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:55.153511 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" event={"ID":"4cf4fae2-dcad-47e2-ae65-d698bb535dab","Type":"ContainerStarted","Data":"9626eb4e6ffb952392289b8d142d17381635fc12dcfa070236db8b3f2066bfdc"} Nov 03 20:12:55.203469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:55.196973 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podStartSLOduration=2.196943736 podStartE2EDuration="2.196943736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:55.195742141 +0000 UTC m=+678.645848188" watchObservedRunningTime="2026-11-03 20:12:55.196943736 +0000 UTC m=+678.647049783" Nov 03 20:12:55.436440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:55.436351 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 03 20:12:55.436440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:55.436408 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 03 20:12:55.988368 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:55.988334 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:56.120410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:56.120377 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:12:56.441288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:56.441254 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:56.441784 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:56.441652 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:56.442479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:56.442432 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 20:12:56.933929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:56.933891 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:57.698036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:57.698003 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:57.989133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:57.989025 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:58.406212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:58.406175 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:58.555394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:58.555358 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:58.710312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:58.710215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:59.123423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:59.121791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:12:59.629372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:12:59.629317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:00.046631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:00.046595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:00.868848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:00.868814 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:01.620012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:01.619969 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:02.655730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:02.655687 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:03.706419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:03.706383 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:04.119992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:04.119960 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:13:04.132167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:04.132141 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:13:04.210829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:04.210801 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:13:04.534181 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:04.534145 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:05.001566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.001493 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq"] Nov 03 20:13:05.005039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.005020 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.011412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.011389 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"kube-root-ca.crt\"" Nov 03 20:13:05.014691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.014669 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq"] Nov 03 20:13:05.015044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.015024 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"openshift-service-ca.crt\"" Nov 03 20:13:05.015352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.015328 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"default-dockercfg-tvmvj\"" Nov 03 20:13:05.015479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.015386 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"simulator-w1-b13a14-kserve-self-signed-certs\"" Nov 03 20:13:05.072991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.072961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:05.162234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.162200 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-dshm\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.162403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.162243 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-home\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.162403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.162309 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pskzh\" (UniqueName: \"kubernetes.io/projected/33a4ea2c-bb5c-4324-994e-564a7a2761a8-kube-api-access-pskzh\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.162403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.162347 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tls-certs\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.162539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.162429 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tmp-dir\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.162539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.162528 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-model-cache\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.263961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.263929 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-home\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.264100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.263976 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pskzh\" (UniqueName: \"kubernetes.io/projected/33a4ea2c-bb5c-4324-994e-564a7a2761a8-kube-api-access-pskzh\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.264100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.264034 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tls-certs\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.264100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.264088 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tmp-dir\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.264228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.264127 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-model-cache\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.264281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.264248 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-dshm\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.264402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.264383 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-home\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.264512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.264488 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tmp-dir\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.264581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.264540 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-model-cache\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.266228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.266197 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-dshm\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.266497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.266480 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tls-certs\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.276310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.276277 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pskzh\" (UniqueName: \"kubernetes.io/projected/33a4ea2c-bb5c-4324-994e-564a7a2761a8-kube-api-access-pskzh\") pod \"simulator-w1-b13a14-kserve-55d5879596-kwgfq\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.305100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.305070 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:05.318565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.318530 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:05.599851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:05.599826 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq"] Nov 03 20:13:05.600887 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:13:05.600861 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33a4ea2c_bb5c_4324_994e_564a7a2761a8.slice/crio-d64a6c19ca64db4b9d4d21a62065f62364174260b6855f8977f1fe024c4fc92f WatchSource:0}: Error finding container d64a6c19ca64db4b9d4d21a62065f62364174260b6855f8977f1fe024c4fc92f: Status 404 returned error can't find the container with id d64a6c19ca64db4b9d4d21a62065f62364174260b6855f8977f1fe024c4fc92f Nov 03 20:13:06.043899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:06.043865 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:06.212592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:06.212553 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" event={"ID":"33a4ea2c-bb5c-4324-994e-564a7a2761a8","Type":"ContainerStarted","Data":"15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e"} Nov 03 20:13:06.212777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:06.212607 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" event={"ID":"33a4ea2c-bb5c-4324-994e-564a7a2761a8","Type":"ContainerStarted","Data":"d64a6c19ca64db4b9d4d21a62065f62364174260b6855f8977f1fe024c4fc92f"} Nov 03 20:13:06.233384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:06.233340 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podStartSLOduration=2.233328597 podStartE2EDuration="2.233328597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:06.231889174 +0000 UTC m=+689.681995220" watchObservedRunningTime="2026-11-03 20:13:06.233328597 +0000 UTC m=+689.683434642" Nov 03 20:13:06.318906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:06.318827 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:06.907187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:06.907151 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:07.603730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:07.603678 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:08.314440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:08.314407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:09.079917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:09.079885 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:09.800634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:09.800591 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:10.499625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.499580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:10.606415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.606384 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs"] Nov 03 20:13:10.610406 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.610384 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.618731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.618703 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"default-dockercfg-qk5xs\"" Nov 03 20:13:10.618887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.618833 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"premium-w3-4f93d1-kserve-self-signed-certs\"" Nov 03 20:13:10.621482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.621457 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs"] Nov 03 20:13:10.627428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.627402 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-home\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.627553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.627462 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-model-cache\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.627599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.627565 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-dshm\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.627651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.627638 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tls-certs\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.627741 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.627721 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tmp-dir\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.627846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.627829 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n87ng\" (UniqueName: \"kubernetes.io/projected/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-kube-api-access-n87ng\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.632019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.631996 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"openshift-service-ca.crt\"" Nov 03 20:13:10.632019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.632002 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"kube-root-ca.crt\"" Nov 03 20:13:10.680597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.680574 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:10.728432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.728408 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-home\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.728581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.728437 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-model-cache\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.728581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.728482 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-dshm\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.728658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.728591 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tls-certs\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.728658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.728631 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tmp-dir\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.728807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.728780 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n87ng\" (UniqueName: \"kubernetes.io/projected/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-kube-api-access-n87ng\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.728865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.728827 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-model-cache\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.728923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.728907 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-home\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.728976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.728966 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tmp-dir\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.730659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.730640 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-dshm\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.730959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.730938 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tls-certs\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.742645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.742616 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n87ng\" (UniqueName: \"kubernetes.io/projected/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-kube-api-access-n87ng\") pod \"premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:10.924156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:10.924119 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:11.204032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:11.204004 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs"] Nov 03 20:13:11.206549 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:13:11.206518 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b9d0cb1_0a0f_497d_a6a2_483f8338466b.slice/crio-870f11e291db86d444cf490d638b4cf3cbb8c226d03b8fabded2be45b3c7204c WatchSource:0}: Error finding container 870f11e291db86d444cf490d638b4cf3cbb8c226d03b8fabded2be45b3c7204c: Status 404 returned error can't find the container with id 870f11e291db86d444cf490d638b4cf3cbb8c226d03b8fabded2be45b3c7204c Nov 03 20:13:11.240794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:11.240762 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" event={"ID":"4b9d0cb1-0a0f-497d-a6a2-483f8338466b","Type":"ContainerStarted","Data":"870f11e291db86d444cf490d638b4cf3cbb8c226d03b8fabded2be45b3c7204c"} Nov 03 20:13:11.497169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:11.497082 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:12.202730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:12.202698 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:12.248294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:12.248251 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" event={"ID":"4b9d0cb1-0a0f-497d-a6a2-483f8338466b","Type":"ContainerStarted","Data":"de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c"} Nov 03 20:13:12.268953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:12.268901 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podStartSLOduration=2.268886688 podStartE2EDuration="2.268886688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:12.266960103 +0000 UTC m=+695.717066147" watchObservedRunningTime="2026-11-03 20:13:12.268886688 +0000 UTC m=+695.718992766" Nov 03 20:13:12.910462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:12.910419 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:12.924439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:12.924407 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:13.706480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:13.706422 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:14.688152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:14.688104 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:15.318972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:15.318934 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:15.331624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:15.331596 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:15.409727 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:15.409696 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:15.984430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:15.984382 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:16.115858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:16.115825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:16.281780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:16.281751 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:13:16.801173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:16.801141 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:17.508531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:17.508497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:17.538061 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:13:17.538024 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:13:18.200491 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:18.200422 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:18.968462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:18.968420 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:19.797929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.797886 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:19.836357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.836327 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh"] Nov 03 20:13:19.840099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.840078 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:19.848004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.847976 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"default-dockercfg-5n6wm\"" Nov 03 20:13:19.848184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.848065 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"kube-root-ca.crt\"" Nov 03 20:13:19.848184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.848131 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"premium-w2-085566-kserve-self-signed-certs\"" Nov 03 20:13:19.848184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.848181 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"openshift-service-ca.crt\"" Nov 03 20:13:19.849588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.849564 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh"] Nov 03 20:13:19.925380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.925344 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:19.930643 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.930619 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-home\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:19.930759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.930732 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01de89c8-912b-4b07-857f-c03e11f08d89-tls-certs\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:19.930801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.930771 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-model-cache\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:19.930835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.930804 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-dshm\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:19.930934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.930916 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-tmp-dir\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:19.930990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:19.930976 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65fk\" (UniqueName: \"kubernetes.io/projected/01de89c8-912b-4b07-857f-c03e11f08d89-kube-api-access-k65fk\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.032118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.032078 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01de89c8-912b-4b07-857f-c03e11f08d89-tls-certs\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.032118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.032123 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-model-cache\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.032328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.032149 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-dshm\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.032328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.032212 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-tmp-dir\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.032328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.032281 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65fk\" (UniqueName: \"kubernetes.io/projected/01de89c8-912b-4b07-857f-c03e11f08d89-kube-api-access-k65fk\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.032328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.032306 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-home\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.032613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.032590 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-model-cache\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.032704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.032642 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-tmp-dir\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.032704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.032680 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-home\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.034593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.034565 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-dshm\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.034725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.034708 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01de89c8-912b-4b07-857f-c03e11f08d89-tls-certs\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.044390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.044369 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65fk\" (UniqueName: \"kubernetes.io/projected/01de89c8-912b-4b07-857f-c03e11f08d89-kube-api-access-k65fk\") pod \"premium-w2-085566-kserve-6c687f56d6-5n2bh\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.154706 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.154616 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:20.435792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.435768 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh"] Nov 03 20:13:20.438068 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:13:20.438035 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01de89c8_912b_4b07_857f_c03e11f08d89.slice/crio-04a83213cbfadf1791418d37204977a0549bf4da2b29706188a0302447ded338 WatchSource:0}: Error finding container 04a83213cbfadf1791418d37204977a0549bf4da2b29706188a0302447ded338: Status 404 returned error can't find the container with id 04a83213cbfadf1791418d37204977a0549bf4da2b29706188a0302447ded338 Nov 03 20:13:20.510944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.510910 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:20.924986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.924956 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:20.937526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:20.937502 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:21.302215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:21.302171 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" event={"ID":"01de89c8-912b-4b07-857f-c03e11f08d89","Type":"ContainerStarted","Data":"6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0"} Nov 03 20:13:21.302418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:21.302226 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" event={"ID":"01de89c8-912b-4b07-857f-c03e11f08d89","Type":"ContainerStarted","Data":"04a83213cbfadf1791418d37204977a0549bf4da2b29706188a0302447ded338"} Nov 03 20:13:21.314113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:21.314080 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:13:21.325461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:21.325399 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podStartSLOduration=2.325386279 podStartE2EDuration="2.325386279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:21.325164425 +0000 UTC m=+704.775270486" watchObservedRunningTime="2026-11-03 20:13:21.325386279 +0000 UTC m=+704.775492326" Nov 03 20:13:21.406681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:21.406643 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:22.109227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:22.109194 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:22.155689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:22.155645 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:22.236655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:22.236617 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:22.909799 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:22.909761 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:23.600683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:23.600641 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:24.289926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:24.289894 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:25.094358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:25.094298 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:25.807060 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:25.807028 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:26.812726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:26.812677 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:27.706455 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:27.706331 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:28.407185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:28.407145 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:29.304520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:29.304474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:30.010408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:30.010370 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:30.155276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:30.155244 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:30.167796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:30.167770 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:30.369388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:30.369289 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:13:30.810843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:30.810807 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:31.519111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:31.519077 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:32.211256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:32.211219 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:32.729668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:32.729634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:32.950703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:32.950668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:33.703987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:33.703948 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:34.522759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:34.522723 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:35.521926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:35.521866 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:36.456482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:36.456383 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:37.382627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:37.382594 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:38.114264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:38.114224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:38.801126 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:38.801092 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:39.664860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:39.664825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:40.295513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:40.295470 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:40.978669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:40.978634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:41.754414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:41.754383 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:42.403585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:42.403546 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:43.101584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:43.101542 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:43.996049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:43.996014 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:44.345779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.345741 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2"] Nov 03 20:13:44.351104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.351085 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.356484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.356434 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3631f-openshift-default-dockercfg-2vnzz\"" Nov 03 20:13:44.365660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.365626 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2"] Nov 03 20:13:44.409226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.409191 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.409226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.409230 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.409430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.409260 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3f57ad-c030-4739-8723-f728de7a129a-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.409430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.409380 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.409430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.409423 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twbmm\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-kube-api-access-twbmm\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.409555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.409525 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3f57ad-c030-4739-8723-f728de7a129a-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.409636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.409622 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.409673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.409648 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.409708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.409684 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.510853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.510799 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.510890 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.510913 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.510939 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3f57ad-c030-4739-8723-f728de7a129a-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.510959 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.510994 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twbmm\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-kube-api-access-twbmm\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.511045 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3f57ad-c030-4739-8723-f728de7a129a-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.511127 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.511155 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.511259 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.511365 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.511496 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.511534 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.511728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.511712 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3f57ad-c030-4739-8723-f728de7a129a-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.513549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.513527 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.513742 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.513717 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3f57ad-c030-4739-8723-f728de7a129a-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.524288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.524264 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twbmm\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-kube-api-access-twbmm\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.524430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.524272 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.665906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.665815 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:44.949834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.949806 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2"] Nov 03 20:13:44.951553 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:13:44.951526 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3f57ad_c030_4739_8723_f728de7a129a.slice/crio-acea8506e17690ba845e7bd0fe24d775d03ea617832ffb65c87fdccca2eee043 WatchSource:0}: Error finding container acea8506e17690ba845e7bd0fe24d775d03ea617832ffb65c87fdccca2eee043: Status 404 returned error can't find the container with id acea8506e17690ba845e7bd0fe24d775d03ea617832ffb65c87fdccca2eee043 Nov 03 20:13:44.954003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.953970 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:44.954091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.954036 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:44.954091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:44.954065 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:45.060633 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:45.060590 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:45.451034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:45.450995 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" event={"ID":"3e3f57ad-c030-4739-8723-f728de7a129a","Type":"ContainerStarted","Data":"b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6"} Nov 03 20:13:45.451221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:45.451091 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" event={"ID":"3e3f57ad-c030-4739-8723-f728de7a129a","Type":"ContainerStarted","Data":"acea8506e17690ba845e7bd0fe24d775d03ea617832ffb65c87fdccca2eee043"} Nov 03 20:13:45.481620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:45.481551 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" podStartSLOduration=1.481530265 podStartE2EDuration="1.481530265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:45.479967047 +0000 UTC m=+728.930073096" watchObservedRunningTime="2026-11-03 20:13:45.481530265 +0000 UTC m=+728.931636316" Nov 03 20:13:45.666553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:45.666520 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:46.018531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:46.018497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:46.692732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:46.692659 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:13:46.693202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:46.692751 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:13:46.814100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:46.814063 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:47.536523 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:13:47.536480 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:13:47.671908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:47.671860 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:13:47.672107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:47.671932 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:13:47.693636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:47.693595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:48.362587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:48.362550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:48.667930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:48.667839 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:13:48.668202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:48.668168 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:13:49.096585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:49.096551 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:49.681943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:49.681894 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:13:49.682250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:49.682222 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:13:49.814306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:49.814273 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:50.620471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:50.609310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:50.667079 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:50.667036 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 20:13:50.667387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:50.667358 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 20:13:51.410299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:51.410263 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:51.671510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:51.671415 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:51.671971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:51.671884 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:51.673312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:51.673287 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:13:52.288142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:52.288106 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:53.332135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:53.332098 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:54.019393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:54.019364 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:54.703316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:54.703279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:55.157553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:55.157515 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:55.504157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:55.504075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:56.206962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:56.206923 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:56.905113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:56.905059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:57.599506 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:57.599472 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:58.292116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:58.292076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:59.021842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.021803 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9"] Nov 03 20:13:59.026078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.026055 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.033109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.033090 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-085566-openshift-default-dockercfg-dt25g\"" Nov 03 20:13:59.039869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.039845 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9"] Nov 03 20:13:59.046196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.046170 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:13:59.209720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.209627 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.209720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.209670 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.209998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.209765 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.209998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.209785 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.209998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.209807 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.209998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.209865 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.209998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.209956 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.210264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.210014 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n267p\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-kube-api-access-n267p\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.210264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.210051 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311230 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311271 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311339 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311359 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311385 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311420 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311475 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311520 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n267p\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-kube-api-access-n267p\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311575 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.311979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311815 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.312073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.311997 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.312073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.312049 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.312224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.312205 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.312566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.312543 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.313890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.313868 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.314124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.314105 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.323834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.323811 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.323962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.323831 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n267p\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-kube-api-access-n267p\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-b6km9\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.342901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.342876 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:13:59.627677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.627648 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9"] Nov 03 20:13:59.629368 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:13:59.629340 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb762bf30_1ddf_4c2c_8d45_c38173b09e5b.slice/crio-d89e07dc5943b9799d53fe9ed6fb4f63995e866a186906db0e9ef0bb57dd6ee7 WatchSource:0}: Error finding container d89e07dc5943b9799d53fe9ed6fb4f63995e866a186906db0e9ef0bb57dd6ee7: Status 404 returned error can't find the container with id d89e07dc5943b9799d53fe9ed6fb4f63995e866a186906db0e9ef0bb57dd6ee7 Nov 03 20:13:59.631731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.631697 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:59.631839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.631774 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:59.631839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:13:59.631815 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:00.070886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:00.070850 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:00.563555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:00.563518 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" event={"ID":"b762bf30-1ddf-4c2c-8d45-c38173b09e5b","Type":"ContainerStarted","Data":"6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c"} Nov 03 20:14:00.563739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:00.563601 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" event={"ID":"b762bf30-1ddf-4c2c-8d45-c38173b09e5b","Type":"ContainerStarted","Data":"d89e07dc5943b9799d53fe9ed6fb4f63995e866a186906db0e9ef0bb57dd6ee7"} Nov 03 20:14:00.607265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:00.607190 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" podStartSLOduration=2.607175248 podStartE2EDuration="2.607175248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:00.604039004 +0000 UTC m=+744.054145049" watchObservedRunningTime="2026-11-03 20:14:00.607175248 +0000 UTC m=+744.057281293" Nov 03 20:14:00.829014 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:00.828907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:01.343795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:01.343728 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:14:01.344714 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:01.344669 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:14:01.344851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:01.344742 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:14:01.729301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:01.729206 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:02.343462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:02.343414 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:14:02.343633 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:02.343502 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:14:02.592021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:02.591979 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:03.343704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:03.343666 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:14:03.343883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:03.343729 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:14:03.541453 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:03.541418 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:04.343291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:04.343254 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:14:04.343802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:04.343319 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:14:04.501779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:04.501652 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:05.312461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:05.312411 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:05.343578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:05.343545 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 20:14:05.343970 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:05.343608 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 20:14:06.089094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:06.089045 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:06.350359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:06.350262 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:14:06.351687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:06.351665 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:14:06.356819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:06.356792 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 20:14:07.136923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:07.136889 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:07.908685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:07.908642 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:08.610142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:08.610104 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:09.335221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:09.335175 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:10.020414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:10.020369 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:10.706506 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:10.706438 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:11.458682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:11.458630 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:12.191511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:12.191475 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:13.113758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:13.113721 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:13.818863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:13.818830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:14.609597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:14.609560 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:15.446323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:15.446281 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:16.281576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:16.281529 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:16.741422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:16.741310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:16.876054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:16.876018 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:17.211118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:17.211060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:17.437514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:17.437482 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:17.532327 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:14:17.532290 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:14:18.274682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:18.274645 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:19.136470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:19.136367 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:20.151498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:20.146306 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:21.318316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:21.318281 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:21.999568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:21.999539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:22.718478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:22.718402 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:23.524308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:23.524272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:24.219242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:24.219204 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:24.914701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:24.914668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:25.656386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:25.656351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:26.566612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:26.566569 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:26.882213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:26.882123 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:27.311429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:27.311392 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:28.093237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:28.093200 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:28.901986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:28.901950 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:29.802422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:29.802390 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:31.252950 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:31.252754 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:32.252467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:32.244888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:33.018858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:33.018825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:33.789740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:33.789707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:34.585833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:34.585796 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:35.297440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.297399 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:35.618337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.618263 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6"] Nov 03 20:14:35.622462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.622430 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.632136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.632112 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6"] Nov 03 20:14:35.632392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.632370 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-466f83\"/\"distinct-w4-466f83-kserve-self-signed-certs\"" Nov 03 20:14:35.684135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.684101 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:35.724375 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.724343 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-tmp-dir\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.724558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.724430 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-model-cache\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.724619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.724594 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s946\" (UniqueName: \"kubernetes.io/projected/57dadd50-7763-424a-91f3-4949d9a7bc79-kube-api-access-9s946\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.724675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.724655 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57dadd50-7763-424a-91f3-4949d9a7bc79-tls-certs\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.724794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.724775 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-dshm\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.724906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.724806 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-home\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.825640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.825603 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s946\" (UniqueName: \"kubernetes.io/projected/57dadd50-7763-424a-91f3-4949d9a7bc79-kube-api-access-9s946\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.825843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.825696 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57dadd50-7763-424a-91f3-4949d9a7bc79-tls-certs\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.825843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.825804 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-dshm\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.825978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.825840 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-home\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.825978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.825959 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-tmp-dir\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.826107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.826068 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-model-cache\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.826470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.826429 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-tmp-dir\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.826596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.826495 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-model-cache\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.826664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.826640 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-home\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.828510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.828485 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-dshm\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.828783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.828762 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57dadd50-7763-424a-91f3-4949d9a7bc79-tls-certs\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.855303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.855276 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s946\" (UniqueName: \"kubernetes.io/projected/57dadd50-7763-424a-91f3-4949d9a7bc79-kube-api-access-9s946\") pod \"distinct-w4-466f83-kserve-6fccd744f4-gwrc6\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:35.936557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:35.936476 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:36.112836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:36.112790 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:36.215530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:36.215504 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6"] Nov 03 20:14:36.217123 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:14:36.217097 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57dadd50_7763_424a_91f3_4949d9a7bc79.slice/crio-4f961dbcd2183ea96fdd61104073edbffa3a1ef1f34137117408d783683147d6 WatchSource:0}: Error finding container 4f961dbcd2183ea96fdd61104073edbffa3a1ef1f34137117408d783683147d6: Status 404 returned error can't find the container with id 4f961dbcd2183ea96fdd61104073edbffa3a1ef1f34137117408d783683147d6 Nov 03 20:14:36.875680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:36.875644 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" event={"ID":"57dadd50-7763-424a-91f3-4949d9a7bc79","Type":"ContainerStarted","Data":"59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3"} Nov 03 20:14:36.876112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:36.875686 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" event={"ID":"57dadd50-7763-424a-91f3-4949d9a7bc79","Type":"ContainerStarted","Data":"4f961dbcd2183ea96fdd61104073edbffa3a1ef1f34137117408d783683147d6"} Nov 03 20:14:36.895933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:36.895869 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podStartSLOduration=1.895855594 podStartE2EDuration="1.895855594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:36.894736283 +0000 UTC m=+780.344842329" watchObservedRunningTime="2026-11-03 20:14:36.895855594 +0000 UTC m=+780.345961639" Nov 03 20:14:36.937589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:36.937552 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:37.221401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:37.221315 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:38.026319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:38.026189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:39.013471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:39.008287 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:40.063042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:40.063005 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:40.859623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:40.859582 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:41.676517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:41.676483 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:42.424807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:42.424771 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:43.209173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:43.209128 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:44.085699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.085661 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:44.438257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.438180 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m"] Nov 03 20:14:44.442770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.442742 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.460174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.460146 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m"] Nov 03 20:14:44.496234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.496195 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.496407 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.496318 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.496407 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.496398 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.496572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.496549 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f21b89-0622-4fc8-84c5-8635f34544ba-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.496636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.496589 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.496726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.496689 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.496788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.496745 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdsl6\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-kube-api-access-zdsl6\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.496844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.496784 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.496902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.496882 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598394 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598471 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598532 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598564 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f21b89-0622-4fc8-84c5-8635f34544ba-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598579 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598601 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598619 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdsl6\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-kube-api-access-zdsl6\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598640 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598665 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598814 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.598991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.598884 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.599156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.599019 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.599156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.599060 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.599270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.599252 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f21b89-0622-4fc8-84c5-8635f34544ba-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.600974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.600927 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.601246 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.601208 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.611039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.611014 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.611129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.611086 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdsl6\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-kube-api-access-zdsl6\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.757931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.757892 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:44.828037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:44.827996 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:45.043943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:45.043915 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m"] Nov 03 20:14:45.045182 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:14:45.045140 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f21b89_0622_4fc8_84c5_8635f34544ba.slice/crio-11213ca392f391eb3c5b992de810e2b4d8faa66c049c9dcc216164679c29abfa WatchSource:0}: Error finding container 11213ca392f391eb3c5b992de810e2b4d8faa66c049c9dcc216164679c29abfa: Status 404 returned error can't find the container with id 11213ca392f391eb3c5b992de810e2b4d8faa66c049c9dcc216164679c29abfa Nov 03 20:14:45.047965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:45.047929 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:45.048040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:45.048011 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:45.048082 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:45.048051 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:45.451313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:45.451222 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:45.937195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:45.937167 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:45.950028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:45.949998 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:45.991752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:45.991655 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" event={"ID":"b8f21b89-0622-4fc8-84c5-8635f34544ba","Type":"ContainerStarted","Data":"094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603"} Nov 03 20:14:45.992100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:45.992080 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" event={"ID":"b8f21b89-0622-4fc8-84c5-8635f34544ba","Type":"ContainerStarted","Data":"11213ca392f391eb3c5b992de810e2b4d8faa66c049c9dcc216164679c29abfa"} Nov 03 20:14:46.003630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:46.003597 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:14:46.016160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:46.016103 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" podStartSLOduration=2.016085852 podStartE2EDuration="2.016085852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:46.015253826 +0000 UTC m=+789.465359895" watchObservedRunningTime="2026-11-03 20:14:46.016085852 +0000 UTC m=+789.466191899" Nov 03 20:14:46.621538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:46.621503 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:46.758118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:46.758087 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:46.759412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:46.759389 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:14:46.759523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:46.759434 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:14:47.425669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:47.425616 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:47.533899 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:14:47.533851 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:14:47.758512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:47.758477 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:14:47.758888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:47.758529 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:14:48.245895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:48.245808 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:48.758569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:48.758538 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:14:48.758961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:48.758586 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:14:49.127275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:49.127193 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:49.758752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:49.758717 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:14:49.759131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:49.758767 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:14:50.011168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:50.011068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:50.700545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:50.700507 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:50.758731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:50.758696 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 20:14:50.758907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:50.758744 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 20:14:51.404437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:51.404401 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:51.762753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:51.762720 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:51.763116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:51.763070 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:51.763769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:51.763751 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:14:52.118162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.118074 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:52.622678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.622637 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2"] Nov 03 20:14:52.627349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.627327 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.633112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.633083 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"e2e-conflict-b-256c5a80-kserve-self-signed-certs\"" Nov 03 20:14:52.634591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.634567 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2"] Nov 03 20:14:52.688524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.688497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:52.717737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.717707 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-dshm\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.717868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.717760 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-tmp-dir\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.717868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.717836 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-model-cache\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.717966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.717866 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvg5\" (UniqueName: \"kubernetes.io/projected/f6091e28-2bfc-41a2-a483-d41b5934561e-kube-api-access-5cvg5\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.717966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.717897 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-home\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.718063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.718046 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6091e28-2bfc-41a2-a483-d41b5934561e-tls-certs\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.819191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.819160 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-dshm\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.819682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.819219 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-tmp-dir\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.819682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.819245 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-model-cache\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.819682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.819263 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvg5\" (UniqueName: \"kubernetes.io/projected/f6091e28-2bfc-41a2-a483-d41b5934561e-kube-api-access-5cvg5\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.819682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.819290 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-home\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.819682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.819384 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6091e28-2bfc-41a2-a483-d41b5934561e-tls-certs\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.819944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.819742 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-tmp-dir\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.819944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.819755 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-model-cache\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.819944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.819860 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-home\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.821414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.821396 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-dshm\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.821902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.821885 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6091e28-2bfc-41a2-a483-d41b5934561e-tls-certs\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.832065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.832041 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvg5\" (UniqueName: \"kubernetes.io/projected/f6091e28-2bfc-41a2-a483-d41b5934561e-kube-api-access-5cvg5\") pod \"e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.943771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.943691 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:52.949577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:52.949550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:53.228317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:53.228293 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2"] Nov 03 20:14:53.229996 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:14:53.229964 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6091e28_2bfc_41a2_a483_d41b5934561e.slice/crio-945faca227512197228543b7d3a135c48aaed32598aefc3310ad79da0274504e WatchSource:0}: Error finding container 945faca227512197228543b7d3a135c48aaed32598aefc3310ad79da0274504e: Status 404 returned error can't find the container with id 945faca227512197228543b7d3a135c48aaed32598aefc3310ad79da0274504e Nov 03 20:14:53.700013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:53.699976 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:54.044399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:54.044365 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" event={"ID":"f6091e28-2bfc-41a2-a483-d41b5934561e","Type":"ContainerStarted","Data":"b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d"} Nov 03 20:14:54.044847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:54.044409 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" event={"ID":"f6091e28-2bfc-41a2-a483-d41b5934561e","Type":"ContainerStarted","Data":"945faca227512197228543b7d3a135c48aaed32598aefc3310ad79da0274504e"} Nov 03 20:14:54.066909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:54.066861 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" podStartSLOduration=2.066848606 podStartE2EDuration="2.066848606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:54.063260795 +0000 UTC m=+797.513366840" watchObservedRunningTime="2026-11-03 20:14:54.066848606 +0000 UTC m=+797.516954650" Nov 03 20:14:54.583192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:54.583162 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:54.944370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:54.944278 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:14:55.224871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:55.224778 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:55.399782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:55.399751 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:56.103887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:56.103841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:56.913797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:56.913748 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:57.703093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:57.703050 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:58.624790 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:58.624748 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:59.191604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:59.191564 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:14:59.579509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:14:59.579464 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:00.152780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.152746 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc"] Nov 03 20:15:00.157013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.156990 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:00.168493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.168463 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc"] Nov 03 20:15:00.170198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.170171 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:15:00.215024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.214993 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dw9\" (UniqueName: \"kubernetes.io/projected/28c8cacf-112f-4508-a29a-93113f67119e-kube-api-access-k4dw9\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc\" (UID: \"28c8cacf-112f-4508-a29a-93113f67119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:00.215171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.215030 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c8cacf-112f-4508-a29a-93113f67119e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc\" (UID: \"28c8cacf-112f-4508-a29a-93113f67119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:00.316083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.316051 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dw9\" (UniqueName: \"kubernetes.io/projected/28c8cacf-112f-4508-a29a-93113f67119e-kube-api-access-k4dw9\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc\" (UID: \"28c8cacf-112f-4508-a29a-93113f67119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:00.316083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.316083 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c8cacf-112f-4508-a29a-93113f67119e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc\" (UID: \"28c8cacf-112f-4508-a29a-93113f67119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:00.316840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.316811 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c8cacf-112f-4508-a29a-93113f67119e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc\" (UID: \"28c8cacf-112f-4508-a29a-93113f67119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:00.330559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.330537 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dw9\" (UniqueName: \"kubernetes.io/projected/28c8cacf-112f-4508-a29a-93113f67119e-kube-api-access-k4dw9\") pod \"maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc\" (UID: \"28c8cacf-112f-4508-a29a-93113f67119e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:00.418646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.418561 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:00.474125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.474095 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:00.752110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:00.752084 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc"] Nov 03 20:15:00.754229 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:15:00.754200 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c8cacf_112f_4508_a29a_93113f67119e.slice/crio-4b8df38e7c7a8577944198faddd6cc0ec5ce2b273b16b36b65a43f906924593e WatchSource:0}: Error finding container 4b8df38e7c7a8577944198faddd6cc0ec5ce2b273b16b36b65a43f906924593e: Status 404 returned error can't find the container with id 4b8df38e7c7a8577944198faddd6cc0ec5ce2b273b16b36b65a43f906924593e Nov 03 20:15:01.083390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:01.083351 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" event={"ID":"28c8cacf-112f-4508-a29a-93113f67119e","Type":"ContainerStarted","Data":"4b8df38e7c7a8577944198faddd6cc0ec5ce2b273b16b36b65a43f906924593e"} Nov 03 20:15:01.108143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:01.108113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:01.513452 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:01.513418 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:01.647125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:01.647093 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:01.914551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:01.914423 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:02.715001 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:02.714959 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:02.944340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:02.944308 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:15:02.956683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:02.956652 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:15:03.111698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:03.111661 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:15:03.566083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:03.566050 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:04.415682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:04.415640 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:05.401434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:05.401335 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:06.131081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:06.130992 4711 generic.go:356] "Generic (PLEG): container finished" podID="28c8cacf-112f-4508-a29a-93113f67119e" containerID="1569e0accfc5f19e9db94c41f34e6af6473e1ec96560fb54fbd71bf3e2d87829" exitCode=60 Nov 03 20:15:06.132548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:06.132500 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" event={"ID":"28c8cacf-112f-4508-a29a-93113f67119e","Type":"ContainerDied","Data":"1569e0accfc5f19e9db94c41f34e6af6473e1ec96560fb54fbd71bf3e2d87829"} Nov 03 20:15:06.132986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:06.132965 4711 scope.go:122] "RemoveContainer" containerID="1569e0accfc5f19e9db94c41f34e6af6473e1ec96560fb54fbd71bf3e2d87829" Nov 03 20:15:07.146062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:07.146019 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" event={"ID":"28c8cacf-112f-4508-a29a-93113f67119e","Type":"ContainerStarted","Data":"441485f2477630ac4a4c0255ecf56b4eb4ee8179dc96c4897d0a3b70c65bf15e"} Nov 03 20:15:07.171471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:07.166027 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" podStartSLOduration=3.098443106 podStartE2EDuration="7.166009794s" totalImagesPullingTime="4.067566688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:07.16422994 +0000 UTC m=+810.614335987" watchObservedRunningTime="2026-11-03 20:15:07.166009794 +0000 UTC m=+810.616115840" Nov 03 20:15:07.191483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:07.190082 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:08.154516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:08.154485 4711 generic.go:356] "Generic (PLEG): container finished" podID="28c8cacf-112f-4508-a29a-93113f67119e" containerID="441485f2477630ac4a4c0255ecf56b4eb4ee8179dc96c4897d0a3b70c65bf15e" exitCode=60 Nov 03 20:15:08.154977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:08.154562 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" event={"ID":"28c8cacf-112f-4508-a29a-93113f67119e","Type":"ContainerDied","Data":"441485f2477630ac4a4c0255ecf56b4eb4ee8179dc96c4897d0a3b70c65bf15e"} Nov 03 20:15:08.154977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:08.154621 4711 scope.go:122] "RemoveContainer" containerID="1569e0accfc5f19e9db94c41f34e6af6473e1ec96560fb54fbd71bf3e2d87829" Nov 03 20:15:08.154977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:08.154847 4711 scope.go:122] "RemoveContainer" containerID="441485f2477630ac4a4c0255ecf56b4eb4ee8179dc96c4897d0a3b70c65bf15e" Nov 03 20:15:08.155150 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:15:08.155131 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc_odh-ai-gateway-infra(28c8cacf-112f-4508-a29a-93113f67119e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" podUID="28c8cacf-112f-4508-a29a-93113f67119e" Nov 03 20:15:08.359531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:08.359498 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:08.814470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:08.809577 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:08.814470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:08.809653 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:08.834254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:08.833631 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:08.834254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:08.833717 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:09.165164 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:09.165034 4711 scope.go:122] "RemoveContainer" containerID="441485f2477630ac4a4c0255ecf56b4eb4ee8179dc96c4897d0a3b70c65bf15e" Nov 03 20:15:09.165846 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:15:09.165377 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc_odh-ai-gateway-infra(28c8cacf-112f-4508-a29a-93113f67119e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" podUID="28c8cacf-112f-4508-a29a-93113f67119e" Nov 03 20:15:09.399542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:09.399505 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:10.149638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:10.149604 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:10.284611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:10.284439 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:11.014197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:11.014160 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:11.790665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:11.790629 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:12.712253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:12.712213 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:13.496561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:13.496521 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:14.192790 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:14.192743 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:14.890759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:14.890720 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:15.356843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:15.356810 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2"] Nov 03 20:15:15.471160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:15.471125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:16.105487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:16.105418 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:16.910427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:16.910343 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:17.152507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:17.152470 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:17.292207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:17.292172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:17.531716 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:15:17.531680 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:15:17.673205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:17.673118 4711 scope.go:122] "RemoveContainer" containerID="441485f2477630ac4a4c0255ecf56b4eb4ee8179dc96c4897d0a3b70c65bf15e" Nov 03 20:15:17.701149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:17.701113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:18.230223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:18.230186 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" event={"ID":"28c8cacf-112f-4508-a29a-93113f67119e","Type":"ContainerStarted","Data":"cfcafe732b640854d7f292d081d60ca8717d22cd19b358f657926c662fb1b394"} Nov 03 20:15:18.499902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:18.497618 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:19.238337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:19.238305 4711 generic.go:356] "Generic (PLEG): container finished" podID="28c8cacf-112f-4508-a29a-93113f67119e" containerID="cfcafe732b640854d7f292d081d60ca8717d22cd19b358f657926c662fb1b394" exitCode=60 Nov 03 20:15:19.238831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:19.238375 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" event={"ID":"28c8cacf-112f-4508-a29a-93113f67119e","Type":"ContainerDied","Data":"cfcafe732b640854d7f292d081d60ca8717d22cd19b358f657926c662fb1b394"} Nov 03 20:15:19.238831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:19.238491 4711 scope.go:122] "RemoveContainer" containerID="441485f2477630ac4a4c0255ecf56b4eb4ee8179dc96c4897d0a3b70c65bf15e" Nov 03 20:15:19.238946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:19.238842 4711 scope.go:122] "RemoveContainer" containerID="cfcafe732b640854d7f292d081d60ca8717d22cd19b358f657926c662fb1b394" Nov 03 20:15:19.239152 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:15:19.239134 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc_odh-ai-gateway-infra(28c8cacf-112f-4508-a29a-93113f67119e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" podUID="28c8cacf-112f-4508-a29a-93113f67119e" Nov 03 20:15:19.257723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:19.257698 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc"] Nov 03 20:15:19.407526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:19.407497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:20.197484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:20.197439 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:20.398984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:20.398960 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:20.503257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:20.503171 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c8cacf-112f-4508-a29a-93113f67119e-kube-api-access-k4dw9\" (UniqueName: \"kubernetes.io/projected/28c8cacf-112f-4508-a29a-93113f67119e-kube-api-access-k4dw9\") pod \"28c8cacf-112f-4508-a29a-93113f67119e\" (UID: \"28c8cacf-112f-4508-a29a-93113f67119e\") " Nov 03 20:15:20.503403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:20.503358 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c8cacf-112f-4508-a29a-93113f67119e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c8cacf-112f-4508-a29a-93113f67119e-maas-api-service-ca\") pod \"28c8cacf-112f-4508-a29a-93113f67119e\" (UID: \"28c8cacf-112f-4508-a29a-93113f67119e\") " Nov 03 20:15:20.503723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:20.503698 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c8cacf-112f-4508-a29a-93113f67119e-maas-api-service-ca" pod "28c8cacf-112f-4508-a29a-93113f67119e" (UID: "28c8cacf-112f-4508-a29a-93113f67119e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.503931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:20.503918 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c8cacf-112f-4508-a29a-93113f67119e-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.505350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:20.505324 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c8cacf-112f-4508-a29a-93113f67119e-kube-api-access-k4dw9" pod "28c8cacf-112f-4508-a29a-93113f67119e" (UID: "28c8cacf-112f-4508-a29a-93113f67119e"). InnerVolumeSpecName "kube-api-access-k4dw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.605429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:20.605387 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4dw9\" (UniqueName: \"kubernetes.io/projected/28c8cacf-112f-4508-a29a-93113f67119e-kube-api-access-k4dw9\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.902498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:20.902412 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:21.255027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:21.255001 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" Nov 03 20:15:21.255202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:21.255030 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc" event={"ID":"28c8cacf-112f-4508-a29a-93113f67119e","Type":"ContainerDied","Data":"4b8df38e7c7a8577944198faddd6cc0ec5ce2b273b16b36b65a43f906924593e"} Nov 03 20:15:21.255202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:21.255092 4711 scope.go:122] "RemoveContainer" containerID="cfcafe732b640854d7f292d081d60ca8717d22cd19b358f657926c662fb1b394" Nov 03 20:15:21.280890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:21.280857 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc"] Nov 03 20:15:21.284646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:21.284624 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4f93d1-29895615-ktjkc"] Nov 03 20:15:21.911144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:21.911107 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:22.685458 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:22.685405 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c8cacf-112f-4508-a29a-93113f67119e" path="/var/lib/kubelet/pods/28c8cacf-112f-4508-a29a-93113f67119e/volumes" Nov 03 20:15:22.701576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:22.701526 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:23.395647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:23.395618 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:23.864549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:23.864517 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:24.220232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:24.220143 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:24.997754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:24.997717 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:25.975982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:25.975938 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:26.773747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:26.773711 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:27.715613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:27.715577 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:28.403216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.403179 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q"] Nov 03 20:15:28.404303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.404277 4711 cpu_manager.go:414] "containerMap: removing container" podUID="28c8cacf-112f-4508-a29a-93113f67119e" containerName="cleanup" Nov 03 20:15:28.404303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.404302 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c8cacf-112f-4508-a29a-93113f67119e" containerName="cleanup" Nov 03 20:15:28.404545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.404315 4711 cpu_manager.go:414] "containerMap: removing container" podUID="28c8cacf-112f-4508-a29a-93113f67119e" containerName="cleanup" Nov 03 20:15:28.404545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.404322 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c8cacf-112f-4508-a29a-93113f67119e" containerName="cleanup" Nov 03 20:15:28.404545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.404371 4711 cpu_manager.go:414] "containerMap: removing container" podUID="28c8cacf-112f-4508-a29a-93113f67119e" containerName="cleanup" Nov 03 20:15:28.404545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.404379 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c8cacf-112f-4508-a29a-93113f67119e" containerName="cleanup" Nov 03 20:15:28.404545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.404518 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c8cacf-112f-4508-a29a-93113f67119e" containerName="cleanup" Nov 03 20:15:28.404545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.404545 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c8cacf-112f-4508-a29a-93113f67119e" containerName="cleanup" Nov 03 20:15:28.405634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.405586 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:28.409577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.409551 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.425651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.425622 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q"] Nov 03 20:15:28.505364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.505326 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.505604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.505393 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.505604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.505540 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.505604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.505583 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.505735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.505698 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.505782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.505755 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.505866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.505850 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.505939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.505886 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4fb\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-kube-api-access-9d4fb\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.505939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.505930 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.542440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.542407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:28.607300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.607248 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.607656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.607631 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.608099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.608063 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.608268 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.608246 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4fb\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-kube-api-access-9d4fb\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.608360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.608314 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.608433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.608417 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.608567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.608524 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.608567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.608561 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.608742 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.608585 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.609357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.609305 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.609821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.609796 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.610050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.610030 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.610146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.610123 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.610282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.610264 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.625343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.625003 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.627886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.627850 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.630065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.629368 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.630065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.630010 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4fb\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-kube-api-access-9d4fb\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.725568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.725482 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:28.756645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.756611 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j"] Nov 03 20:15:28.757554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.757534 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c8cacf-112f-4508-a29a-93113f67119e" containerName="cleanup" Nov 03 20:15:28.762069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.762050 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.763991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.763964 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:28.767341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.767264 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fe792c5024bb\"" Nov 03 20:15:28.767460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.767389 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4a6620c49c4f\"" Nov 03 20:15:28.767567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.767551 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7ddlc\"" Nov 03 20:15:28.769553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.768109 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c1efc772c942\"" Nov 03 20:15:28.769553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.769488 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 20:15:28.769553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.769571 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 20:15:28.769553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.769656 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 20:15:28.769553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.769709 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 20:15:28.771963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.771764 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j"] Nov 03 20:15:28.819725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.819236 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-provider-credentials\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.821969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.821615 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-plugins-config-volume\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.821969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.821687 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bzv\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-kube-api-access-c8bzv\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.821969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.821750 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-routing-overlay-volume\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.922752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.922707 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bzv\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-kube-api-access-c8bzv\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.922929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.922764 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-routing-overlay-volume\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.922929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.922881 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-provider-credentials\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.923045 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.923032 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-plugins-config-volume\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.923462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.923429 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-routing-overlay-volume\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.923699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.923683 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-plugins-config-volume\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.925425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.925404 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-provider-credentials\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:28.937405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:28.937376 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bzv\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-kube-api-access-c8bzv\") pod \"payload-processing-external-model-5cfccf547d-qz86j\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:29.056489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.056458 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q"] Nov 03 20:15:29.057638 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:15:29.057611 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8f34d6_5147_405a_8c9f_ea912c4b6b34.slice/crio-5da5e93a41774391ba45c1806ce4b8421439489be9b77f79b137f6c3a2ec791e WatchSource:0}: Error finding container 5da5e93a41774391ba45c1806ce4b8421439489be9b77f79b137f6c3a2ec791e: Status 404 returned error can't find the container with id 5da5e93a41774391ba45c1806ce4b8421439489be9b77f79b137f6c3a2ec791e Nov 03 20:15:29.060232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.060194 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:29.060299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.060267 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:29.060335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.060297 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:29.091300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.091276 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:29.312244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.312150 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" event={"ID":"7c8f34d6-5147-405a-8c9f-ea912c4b6b34","Type":"ContainerStarted","Data":"2160c3c559c1b62b57b8a5b118ca39b4575d3308f958447257be04a1bf385306"} Nov 03 20:15:29.312244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.312203 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" event={"ID":"7c8f34d6-5147-405a-8c9f-ea912c4b6b34","Type":"ContainerStarted","Data":"5da5e93a41774391ba45c1806ce4b8421439489be9b77f79b137f6c3a2ec791e"} Nov 03 20:15:29.345042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.344982 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podStartSLOduration=1.344962874 podStartE2EDuration="1.344962874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:29.34150886 +0000 UTC m=+832.791614907" watchObservedRunningTime="2026-11-03 20:15:29.344962874 +0000 UTC m=+832.795068922" Nov 03 20:15:29.385167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.385136 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j"] Nov 03 20:15:29.387273 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:15:29.387242 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9bbc78_f089_44f5_ab9e_3a986df305d1.slice/crio-476bb11b481b0f289c3ec55df210fb34e0ba6059b90d98706583a706e2836633 WatchSource:0}: Error finding container 476bb11b481b0f289c3ec55df210fb34e0ba6059b90d98706583a706e2836633: Status 404 returned error can't find the container with id 476bb11b481b0f289c3ec55df210fb34e0ba6059b90d98706583a706e2836633 Nov 03 20:15:29.725842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:29.725764 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:30.031616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.031585 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:30.320693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.320603 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" event={"ID":"bc9bbc78-f089-44f5-ab9e-3a986df305d1","Type":"ContainerStarted","Data":"80f6d0bbbbd447d72bfe60a374dbaf182484d517697b0f21590e14297cc854d1"} Nov 03 20:15:30.320693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.320660 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" event={"ID":"bc9bbc78-f089-44f5-ab9e-3a986df305d1","Type":"ContainerStarted","Data":"476bb11b481b0f289c3ec55df210fb34e0ba6059b90d98706583a706e2836633"} Nov 03 20:15:30.320916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.320892 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:30.341249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.341193 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" podStartSLOduration=2.341178082 podStartE2EDuration="2.341178082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:30.339382495 +0000 UTC m=+833.789488541" watchObservedRunningTime="2026-11-03 20:15:30.341178082 +0000 UTC m=+833.791284154" Nov 03 20:15:30.390875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.390782 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" podUID="f6091e28-2bfc-41a2-a483-d41b5934561e" containerName="main" containerID="cri-o://b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d" gracePeriod=45 Nov 03 20:15:30.704915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.704889 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:15:30.728234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.728199 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:15:30.728603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.728254 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:15:30.747006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.746969 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-tmp-dir\") pod \"f6091e28-2bfc-41a2-a483-d41b5934561e\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " Nov 03 20:15:30.747156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.747035 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6091e28-2bfc-41a2-a483-d41b5934561e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6091e28-2bfc-41a2-a483-d41b5934561e-tls-certs\") pod \"f6091e28-2bfc-41a2-a483-d41b5934561e\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " Nov 03 20:15:30.747156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.747075 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-home\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-home\") pod \"f6091e28-2bfc-41a2-a483-d41b5934561e\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " Nov 03 20:15:30.747156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.747125 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-dshm\") pod \"f6091e28-2bfc-41a2-a483-d41b5934561e\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " Nov 03 20:15:30.747321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.747201 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-model-cache\") pod \"f6091e28-2bfc-41a2-a483-d41b5934561e\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " Nov 03 20:15:30.747370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.747325 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-tmp-dir" pod "f6091e28-2bfc-41a2-a483-d41b5934561e" (UID: "f6091e28-2bfc-41a2-a483-d41b5934561e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:30.747370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.747344 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6091e28-2bfc-41a2-a483-d41b5934561e-kube-api-access-5cvg5\" (UniqueName: \"kubernetes.io/projected/f6091e28-2bfc-41a2-a483-d41b5934561e-kube-api-access-5cvg5\") pod \"f6091e28-2bfc-41a2-a483-d41b5934561e\" (UID: \"f6091e28-2bfc-41a2-a483-d41b5934561e\") " Nov 03 20:15:30.747484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.747361 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-home" pod "f6091e28-2bfc-41a2-a483-d41b5934561e" (UID: "f6091e28-2bfc-41a2-a483-d41b5934561e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:30.747624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.747604 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-model-cache" pod "f6091e28-2bfc-41a2-a483-d41b5934561e" (UID: "f6091e28-2bfc-41a2-a483-d41b5934561e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:30.748135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.748108 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:30.748249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.748138 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:30.748249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.748153 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:30.749668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.749646 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-dshm" pod "f6091e28-2bfc-41a2-a483-d41b5934561e" (UID: "f6091e28-2bfc-41a2-a483-d41b5934561e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:15:30.749775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.749720 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6091e28-2bfc-41a2-a483-d41b5934561e-tls-certs" pod "f6091e28-2bfc-41a2-a483-d41b5934561e" (UID: "f6091e28-2bfc-41a2-a483-d41b5934561e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:30.750212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.750187 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6091e28-2bfc-41a2-a483-d41b5934561e-kube-api-access-5cvg5" pod "f6091e28-2bfc-41a2-a483-d41b5934561e" (UID: "f6091e28-2bfc-41a2-a483-d41b5934561e"). InnerVolumeSpecName "kube-api-access-5cvg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:30.849615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.849585 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvg5\" (UniqueName: \"kubernetes.io/projected/f6091e28-2bfc-41a2-a483-d41b5934561e-kube-api-access-5cvg5\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:30.849615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.849612 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6091e28-2bfc-41a2-a483-d41b5934561e-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:30.849615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:30.849622 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6091e28-2bfc-41a2-a483-d41b5934561e-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:31.102930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.102888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:31.328144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.328108 4711 generic.go:356] "Generic (PLEG): container finished" podID="f6091e28-2bfc-41a2-a483-d41b5934561e" containerID="b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d" exitCode=0 Nov 03 20:15:31.328314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.328160 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" Nov 03 20:15:31.328314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.328181 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" event={"ID":"f6091e28-2bfc-41a2-a483-d41b5934561e","Type":"ContainerDied","Data":"b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d"} Nov 03 20:15:31.328314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.328218 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2" event={"ID":"f6091e28-2bfc-41a2-a483-d41b5934561e","Type":"ContainerDied","Data":"945faca227512197228543b7d3a135c48aaed32598aefc3310ad79da0274504e"} Nov 03 20:15:31.328314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.328234 4711 scope.go:122] "RemoveContainer" containerID="b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d" Nov 03 20:15:31.339078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.339061 4711 scope.go:122] "RemoveContainer" containerID="b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d" Nov 03 20:15:31.339355 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:15:31.339337 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d\": container with ID starting with b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d not found: ID does not exist" containerID="b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d" Nov 03 20:15:31.339425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.339360 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d"} err="failed to get container status \"b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d\": rpc error: code = NotFound desc = could not find container \"b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d\": container with ID starting with b3df1fbee05171ed02a361022d3f7e960dc288766dd0c89b23673074ae28609d not found: ID does not exist" Nov 03 20:15:31.354227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.354171 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2"] Nov 03 20:15:31.358437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.358415 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/e2e-conflict-b-256c5a80-kserve-57b474fbcc-mb7t2"] Nov 03 20:15:31.726890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.726806 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:15:31.726890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.726861 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:15:31.930199 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:31.930167 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:32.697344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:32.697298 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6091e28-2bfc-41a2-a483-d41b5934561e" path="/var/lib/kubelet/pods/f6091e28-2bfc-41a2-a483-d41b5934561e/volumes" Nov 03 20:15:32.718475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:32.718429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:32.726592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:32.726568 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:15:32.726723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:32.726616 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:15:33.583493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:33.573889 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:33.726353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:33.726318 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:15:33.726559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:33.726376 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:15:34.320199 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:34.320161 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:34.726174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:34.726094 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:15:34.726174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:34.726159 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:15:35.253190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:35.253149 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:35.726664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:35.726630 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 20:15:35.727073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:35.726703 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 20:15:36.153087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:36.153050 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:37.780466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:37.770745 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:37.780466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:37.770810 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:37.952468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:37.951196 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:37.995584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:37.995537 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:38.207599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:38.207488 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:38.272895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:38.272853 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:15:38.317469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:38.316852 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:38.321581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:38.317888 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:38.370587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:38.370543 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:38.821604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:38.821562 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:38.823900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:38.823159 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:39.059603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.059555 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:39.065474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.060578 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:15:39.286471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.279274 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:39.286471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.279519 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:15:39.406471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.348554 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:15:39.406471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.349390 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:15:39.406471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.377495 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:39.406471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.378186 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:15:39.808498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.787527 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24"] Nov 03 20:15:39.808498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.788424 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f6091e28-2bfc-41a2-a483-d41b5934561e" containerName="main" Nov 03 20:15:39.808498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.788462 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6091e28-2bfc-41a2-a483-d41b5934561e" containerName="main" Nov 03 20:15:39.808498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.788631 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6091e28-2bfc-41a2-a483-d41b5934561e" containerName="main" Nov 03 20:15:39.808498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.796066 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.899478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.898840 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.899478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.898902 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.899478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.898986 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.899478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.899021 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8nmg\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-kube-api-access-j8nmg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.899478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.899071 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.899478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.899099 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031104d1-af70-42bc-b158-659f3b3a2f37-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.899478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.899197 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031104d1-af70-42bc-b158-659f3b3a2f37-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.899478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.899250 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.899478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.899348 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:39.900415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.900294 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:39.900415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.900342 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:40.001497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.942217 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24"] Nov 03 20:15:40.001497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.950768 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:40.001497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.952256 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:40.001497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.956541 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:15:40.001497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:39.998111 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.003549 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.003699 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.003776 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.003820 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.003928 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.003973 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8nmg\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-kube-api-access-j8nmg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.004055 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.004107 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031104d1-af70-42bc-b158-659f3b3a2f37-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.004270 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031104d1-af70-42bc-b158-659f3b3a2f37-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.014533 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.015378 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.050470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.031544 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031104d1-af70-42bc-b158-659f3b3a2f37-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.098526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.061580 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:40.098526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.063682 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.098526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.064402 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.146379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.146340 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031104d1-af70-42bc-b158-659f3b3a2f37-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.147143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.147114 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.240079 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.238554 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:15:40.261778 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.243950 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:15:40.289479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.287099 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:15:40.335499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.319077 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:15:40.335499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.321912 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8nmg\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-kube-api-access-j8nmg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.459219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.459137 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:15:40.559190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.559152 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:40.560797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.559937 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:40.719040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:40.718951 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:41.333470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:41.333023 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:15:41.430253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:41.430220 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24"] Nov 03 20:15:41.432104 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:15:41.432076 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031104d1_af70_42bc_b158_659f3b3a2f37.slice/crio-70596605e92ea2d0c36002d2b04a6a7cad86dd40fadab8239ca896c6f6ff14a9 WatchSource:0}: Error finding container 70596605e92ea2d0c36002d2b04a6a7cad86dd40fadab8239ca896c6f6ff14a9: Status 404 returned error can't find the container with id 70596605e92ea2d0c36002d2b04a6a7cad86dd40fadab8239ca896c6f6ff14a9 Nov 03 20:15:41.434910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:41.434871 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:41.435023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:41.434976 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:41.435094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:41.435021 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:41.665330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:41.665295 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" event={"ID":"031104d1-af70-42bc-b158-659f3b3a2f37","Type":"ContainerStarted","Data":"62b54c29e2596c5868e66558dc6d3a13f5511aef75930d69169b5cb18b086004"} Nov 03 20:15:41.665554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:41.665351 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" event={"ID":"031104d1-af70-42bc-b158-659f3b3a2f37","Type":"ContainerStarted","Data":"70596605e92ea2d0c36002d2b04a6a7cad86dd40fadab8239ca896c6f6ff14a9"} Nov 03 20:15:41.719953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:41.719864 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:42.038373 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:42.038335 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:42.721861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:42.721817 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:15:42.722267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:42.721883 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:15:43.720123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:43.720082 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:15:43.720303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:43.720142 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:15:44.499843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.499787 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" podStartSLOduration=5.499770833 podStartE2EDuration="5.499770833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:41.689311309 +0000 UTC m=+845.139417356" watchObservedRunningTime="2026-11-03 20:15:44.499770833 +0000 UTC m=+847.949876879" Nov 03 20:15:44.501829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.501797 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4"] Nov 03 20:15:44.507830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.507783 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4"] Nov 03 20:15:44.507965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.507903 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.511933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.511907 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4f93d1\"/\"distinct-2-w3-4f93d1-kserve-self-signed-certs\"" Nov 03 20:15:44.624577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.623499 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-dshm\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.624577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.623574 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5nsz\" (UniqueName: \"kubernetes.io/projected/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-kube-api-access-t5nsz\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.624577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.623668 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tmp-dir\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.624577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.623779 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-home\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.624577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.623820 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tls-certs\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.624577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.623892 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-model-cache\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.719731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.719700 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:15:44.719897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.719750 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:15:44.725521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.725497 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tls-certs\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.725608 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.725559 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-model-cache\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.725670 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.725654 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-dshm\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.725713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.725678 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5nsz\" (UniqueName: \"kubernetes.io/projected/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-kube-api-access-t5nsz\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.725755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.725735 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tmp-dir\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.725836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.725821 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-home\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.726111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.726087 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-model-cache\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.726111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.726100 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-home\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.726239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.726193 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tmp-dir\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.727905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.727874 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-dshm\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.728010 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.727950 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tls-certs\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.737488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.737438 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5nsz\" (UniqueName: \"kubernetes.io/projected/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-kube-api-access-t5nsz\") pod \"distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:44.817929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.817883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:44.821826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:44.821795 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:45.132288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:45.132257 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4"] Nov 03 20:15:45.134007 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:15:45.133980 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ad5f3f_8ae3_4c11_804f_2eba281ec11d.slice/crio-ec7541bf82290ae8976d7c6a286b8ba313e8d68c0ef39de8945648f3400637c4 WatchSource:0}: Error finding container ec7541bf82290ae8976d7c6a286b8ba313e8d68c0ef39de8945648f3400637c4: Status 404 returned error can't find the container with id ec7541bf82290ae8976d7c6a286b8ba313e8d68c0ef39de8945648f3400637c4 Nov 03 20:15:45.215701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:45.215668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:45.720294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:45.720254 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:15:45.720746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:45.720325 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:15:45.727971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:45.727931 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" event={"ID":"55ad5f3f-8ae3-4c11-804f-2eba281ec11d","Type":"ContainerStarted","Data":"fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7"} Nov 03 20:15:45.728143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:45.727986 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" event={"ID":"55ad5f3f-8ae3-4c11-804f-2eba281ec11d","Type":"ContainerStarted","Data":"ec7541bf82290ae8976d7c6a286b8ba313e8d68c0ef39de8945648f3400637c4"} Nov 03 20:15:45.747346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:45.747301 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podStartSLOduration=1.7472854820000001 podStartE2EDuration="1.747285482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:45.746302562 +0000 UTC m=+849.196408647" watchObservedRunningTime="2026-11-03 20:15:45.747285482 +0000 UTC m=+849.197391528" Nov 03 20:15:45.822921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:45.822883 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:45.895572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:45.895535 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:46.704085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:46.704048 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:46.720232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:46.720203 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:15:46.720382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:46.720255 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:15:47.546594 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:15:47.546549 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:15:47.562796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:47.562762 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:47.720572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:47.720538 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 20:15:47.720977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:47.720594 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 20:15:48.198038 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:48.197997 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:48.724372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:48.724341 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:48.725030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:48.724857 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:48.725705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:48.725687 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:15:48.904900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:48.904863 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:49.602243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:49.602205 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:50.257659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:50.257615 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:50.945670 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:50.945627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:51.618599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:51.618565 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:52.487159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:52.487119 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:54.208470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:54.207872 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:15:54.304471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:54.303427 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:54.304471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:54.303507 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:54.326472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:54.325300 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:15:54.342841 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:54.342533 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:54.831866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:54.831821 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:54.945943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:54.945896 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:55.518902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:55.518864 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:55.840088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:55.839997 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:55.865209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:55.865177 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:15:57.110687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:57.110650 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-947bh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:57.111267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:57.111216 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:57.253083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:57.252993 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:58.215024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:58.214989 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:59.089821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:59.089768 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:15:59.822557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:15:59.822519 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:00.591343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:00.591306 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:01.466501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:01.466466 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:02.402650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:02.402617 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:03.370632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:03.370595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:04.441354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:04.441317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:05.431176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:05.431141 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:06.426430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:06.426395 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:06.565383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:06.565351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:08.450501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:08.450431 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:09.928562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:09.928524 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:10.243090 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.243003 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6"] Nov 03 20:16:10.247677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.247656 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.260035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.259883 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6"] Nov 03 20:16:10.260563 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.260463 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:10.276628 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.276597 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-provider-credentials\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.276806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.276697 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-routing-overlay-volume\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.276912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.276884 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-plugins-config-volume\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.277015 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.276997 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7hn5\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-kube-api-access-m7hn5\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.391484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.391433 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:10.395282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.395241 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7hn5\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-kube-api-access-m7hn5\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.395493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.395388 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-provider-credentials\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.395493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.395461 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-routing-overlay-volume\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.395600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.395582 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-plugins-config-volume\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.396590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.396537 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-plugins-config-volume\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.405383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.405352 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-provider-credentials\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.406227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.406204 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-routing-overlay-volume\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.411826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.411796 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7hn5\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-kube-api-access-m7hn5\") pod \"payload-processing-external-model-9dbdc456b-mjdh6\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.566717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.566687 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:10.855730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:10.855703 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6"] Nov 03 20:16:10.857353 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:16:10.857329 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318d5333_5221_4001_893c_fc1162d1fda4.slice/crio-b7e5e35e073bf990e0b7b295b98a169aaaedcaf16a89f6b2bcc4e293a4a33c89 WatchSource:0}: Error finding container b7e5e35e073bf990e0b7b295b98a169aaaedcaf16a89f6b2bcc4e293a4a33c89: Status 404 returned error can't find the container with id b7e5e35e073bf990e0b7b295b98a169aaaedcaf16a89f6b2bcc4e293a4a33c89 Nov 03 20:16:11.072598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:11.072564 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" event={"ID":"318d5333-5221-4001-893c-fc1162d1fda4","Type":"ContainerStarted","Data":"78654cb0827dc31393149f4c0e5202f1314fe4fd56547f87f1d41c52bc635a62"} Nov 03 20:16:11.073081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:11.072627 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:11.073081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:11.072651 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" event={"ID":"318d5333-5221-4001-893c-fc1162d1fda4","Type":"ContainerStarted","Data":"b7e5e35e073bf990e0b7b295b98a169aaaedcaf16a89f6b2bcc4e293a4a33c89"} Nov 03 20:16:11.095931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:11.095882 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" podStartSLOduration=1.095868145 podStartE2EDuration="1.095868145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:11.091979807 +0000 UTC m=+874.542085853" watchObservedRunningTime="2026-11-03 20:16:11.095868145 +0000 UTC m=+874.545974227" Nov 03 20:16:11.383411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:11.383374 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:12.214004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:12.213966 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:12.971753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:12.971720 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:13.469789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.469753 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv"] Nov 03 20:16:13.474029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.474009 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.493318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.493288 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv"] Nov 03 20:16:13.532229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.532193 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21270674-676b-4739-b6ed-dda82d8b1a22-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.532229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.532232 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.532488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.532393 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.532488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.532432 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.532488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.532476 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8xt\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-kube-api-access-ml8xt\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.532668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.532582 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.532668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.532662 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.532803 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.532693 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21270674-676b-4739-b6ed-dda82d8b1a22-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.532803 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.532773 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.634301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634263 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.634301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634303 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.634683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634318 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8xt\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-kube-api-access-ml8xt\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.634683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634369 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.634683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634459 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.634683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634475 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21270674-676b-4739-b6ed-dda82d8b1a22-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.634683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634493 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.634683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634582 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21270674-676b-4739-b6ed-dda82d8b1a22-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.634683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634614 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.635050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634874 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-certs\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.635050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634953 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-credential-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.635050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.634993 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-data\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.635050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.635041 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-socket\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.635303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.635283 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21270674-676b-4739-b6ed-dda82d8b1a22-istiod-ca-cert\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.637191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.637169 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-envoy\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.638113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.638073 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21270674-676b-4739-b6ed-dda82d8b1a22-istio-podinfo\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.647581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.647551 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-istio-token\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.647695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.647676 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8xt\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-kube-api-access-ml8xt\") pod \"e2e-worker-w4-466f83-openshift-default-5559865c55-546kv\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.788922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.788887 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:13.868651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:13.868600 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:14.073786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.073756 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv"] Nov 03 20:16:14.074541 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:16:14.074513 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21270674_676b_4739_b6ed_dda82d8b1a22.slice/crio-b58d0b26b16f91d642685421e1a91641c17298bb30e213a73b3fe90ed57d3d61 WatchSource:0}: Error finding container b58d0b26b16f91d642685421e1a91641c17298bb30e213a73b3fe90ed57d3d61: Status 404 returned error can't find the container with id b58d0b26b16f91d642685421e1a91641c17298bb30e213a73b3fe90ed57d3d61 Nov 03 20:16:14.077232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.077203 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:14.077313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.077266 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:14.077313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.077299 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:14.093273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.093247 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" event={"ID":"21270674-676b-4739-b6ed-dda82d8b1a22","Type":"ContainerStarted","Data":"b58d0b26b16f91d642685421e1a91641c17298bb30e213a73b3fe90ed57d3d61"} Nov 03 20:16:14.556931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.556903 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz"] Nov 03 20:16:14.561368 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.561346 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.608918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.606365 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz"] Nov 03 20:16:14.648850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.648817 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eef084f-fa41-48aa-becf-55d52edd3050-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.649016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.648853 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvwx8\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-kube-api-access-pvwx8\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.649016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.648922 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.649016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.648962 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.649195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.649025 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.649195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.649058 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.649195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.649166 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.649355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.649206 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eef084f-fa41-48aa-becf-55d52edd3050-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.649355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.649335 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.721470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.721424 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:14.750423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.750388 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eef084f-fa41-48aa-becf-55d52edd3050-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.750636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.750431 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvwx8\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-kube-api-access-pvwx8\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.750636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.750478 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.750636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.750539 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.750636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.750597 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.750863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.750640 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.750863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.750815 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.750969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.750901 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eef084f-fa41-48aa-becf-55d52edd3050-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.751025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.751005 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.751224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.751203 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eef084f-fa41-48aa-becf-55d52edd3050-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.751331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.751252 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.751331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.751261 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.751331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.751287 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.751708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.751678 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.753586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.753565 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.754336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.754318 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eef084f-fa41-48aa-becf-55d52edd3050-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.763781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.763750 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.764957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.764934 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvwx8\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-kube-api-access-pvwx8\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:14.909765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:14.909673 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:15.006389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.006331 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr"] Nov 03 20:16:15.013831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.013810 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr"] Nov 03 20:16:15.014155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.014088 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.024138 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.024114 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b13a14\"/\"distinct-w1-b13a14-kserve-self-signed-certs\"" Nov 03 20:16:15.048751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.048636 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:15.113895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.113731 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" event={"ID":"21270674-676b-4739-b6ed-dda82d8b1a22","Type":"ContainerStarted","Data":"87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824"} Nov 03 20:16:15.143501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.143182 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" podStartSLOduration=2.143163247 podStartE2EDuration="2.143163247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:15.142805922 +0000 UTC m=+878.592911968" watchObservedRunningTime="2026-11-03 20:16:15.143163247 +0000 UTC m=+878.593269293" Nov 03 20:16:15.157376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.157345 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-home\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.157557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.157381 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvxt9\" (UniqueName: \"kubernetes.io/projected/fda8d702-b23b-4df0-8540-febd71b7f7fa-kube-api-access-lvxt9\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.157657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.157614 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-tmp-dir\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.157739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.157723 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-dshm\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.157800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.157785 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda8d702-b23b-4df0-8540-febd71b7f7fa-tls-certs\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.157845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.157829 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-model-cache\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.237027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.236996 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz"] Nov 03 20:16:15.238172 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:16:15.238150 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eef084f_fa41_48aa_becf_55d52edd3050.slice/crio-47d4de77c9cab2293bc29517865ac78a9ebed315401a1bd88dd59b60a9f2618f WatchSource:0}: Error finding container 47d4de77c9cab2293bc29517865ac78a9ebed315401a1bd88dd59b60a9f2618f: Status 404 returned error can't find the container with id 47d4de77c9cab2293bc29517865ac78a9ebed315401a1bd88dd59b60a9f2618f Nov 03 20:16:15.240823 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.240784 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:15.240923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.240863 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:15.240923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.240899 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:15.258747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.258716 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-model-cache\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.258869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.258843 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-home\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.258928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.258870 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvxt9\" (UniqueName: \"kubernetes.io/projected/fda8d702-b23b-4df0-8540-febd71b7f7fa-kube-api-access-lvxt9\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.259050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.259026 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-tmp-dir\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.259108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.259049 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-model-cache\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.259425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.259398 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-dshm\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.259590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.259505 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda8d702-b23b-4df0-8540-febd71b7f7fa-tls-certs\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.259590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.259547 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-tmp-dir\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.259757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.259735 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-home\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.261785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.261769 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-dshm\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.261950 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.261933 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda8d702-b23b-4df0-8540-febd71b7f7fa-tls-certs\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.301845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.298810 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvxt9\" (UniqueName: \"kubernetes.io/projected/fda8d702-b23b-4df0-8540-febd71b7f7fa-kube-api-access-lvxt9\") pod \"distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.333646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.333618 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:15.649499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.649462 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr"] Nov 03 20:16:15.651058 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:16:15.651010 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfda8d702_b23b_4df0_8540_febd71b7f7fa.slice/crio-14f55123187a63c974bec758253ee38673b23aae12233656048892713401b1b6 WatchSource:0}: Error finding container 14f55123187a63c974bec758253ee38673b23aae12233656048892713401b1b6: Status 404 returned error can't find the container with id 14f55123187a63c974bec758253ee38673b23aae12233656048892713401b1b6 Nov 03 20:16:15.710220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.710189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:15.789693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.789666 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:15.792428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.792401 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:15.792663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:15.792629 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 20:16:16.126571 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.126537 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" event={"ID":"fda8d702-b23b-4df0-8540-febd71b7f7fa","Type":"ContainerStarted","Data":"38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9"} Nov 03 20:16:16.126571 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.126578 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" event={"ID":"fda8d702-b23b-4df0-8540-febd71b7f7fa","Type":"ContainerStarted","Data":"14f55123187a63c974bec758253ee38673b23aae12233656048892713401b1b6"} Nov 03 20:16:16.129601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.129575 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" event={"ID":"1eef084f-fa41-48aa-becf-55d52edd3050","Type":"ContainerStarted","Data":"f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957"} Nov 03 20:16:16.129740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.129611 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" event={"ID":"1eef084f-fa41-48aa-becf-55d52edd3050","Type":"ContainerStarted","Data":"47d4de77c9cab2293bc29517865ac78a9ebed315401a1bd88dd59b60a9f2618f"} Nov 03 20:16:16.154764 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.154724 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podStartSLOduration=2.154709827 podStartE2EDuration="2.154709827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:16.151876073 +0000 UTC m=+879.601982132" watchObservedRunningTime="2026-11-03 20:16:16.154709827 +0000 UTC m=+879.604815866" Nov 03 20:16:16.210821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.210754 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" podStartSLOduration=2.210736154 podStartE2EDuration="2.210736154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:16.208391382 +0000 UTC m=+879.658497430" watchObservedRunningTime="2026-11-03 20:16:16.210736154 +0000 UTC m=+879.660842203" Nov 03 20:16:16.334566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.334533 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:16.799875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.799837 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:16.800498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.800437 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 20:16:16.818203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.818080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:16.919502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.915822 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:16.919502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.918566 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:16.919502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:16.918616 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:16:17.548477 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:16:17.548431 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:16:17.812009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:17.811898 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:17.812009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:17.811986 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 20:16:17.915836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:17.915788 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:17.916007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:17.915847 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:16:17.917965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:17.917937 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:18.806471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:18.793409 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:18.806471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:18.793495 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 20:16:18.913466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:18.912871 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:18.913466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:18.912936 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:16:18.957797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:18.957763 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:19.790469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:19.789780 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:19.790469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:19.789842 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 20:16:19.915821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:19.915787 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:19.916256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:19.916232 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:16:20.001147 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:20.001102 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:20.790468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:20.789804 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 20:16:20.790468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:20.789866 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 20:16:20.910988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:20.910952 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:20.911164 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:20.911020 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:16:20.928072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:20.928025 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:21.794158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:21.794125 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:21.795268 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:21.795245 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:21.795835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:21.795814 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:16:21.900736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:21.900683 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:21.911455 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:21.911414 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:16:21.911673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:21.911644 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:16:22.086481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:22.086382 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:22.147753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:22.147721 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j"] Nov 03 20:16:22.297278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:22.297207 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:22.450031 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:22.431761 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6"] Nov 03 20:16:22.450031 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:22.434589 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:22.576562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:22.576526 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:22.916189 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:22.916151 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:22.916572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:22.916548 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:22.917249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:22.917231 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:16:23.114128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:23.114087 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:24.098697 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.098658 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:24.817547 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.817508 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp"] Nov 03 20:16:24.821764 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.821735 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.851364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.851332 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp"] Nov 03 20:16:24.873725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.873690 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.873976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.873732 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adfd185-9df3-44db-afda-b5c67a858d54-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.873976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.873766 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.874197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.873964 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghpbj\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-kube-api-access-ghpbj\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.874197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.874104 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.874197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.874168 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.874368 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.874333 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.874512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.874437 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.874512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.874495 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adfd185-9df3-44db-afda-b5c67a858d54-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.920555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.920513 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:24.975709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.975677 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.975926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.975763 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adfd185-9df3-44db-afda-b5c67a858d54-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.975926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.975862 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.975926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.975886 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adfd185-9df3-44db-afda-b5c67a858d54-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.975926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.975920 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.976173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.975959 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghpbj\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-kube-api-access-ghpbj\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.976173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.976039 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.976173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.976109 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.976173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.976139 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.976428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.976228 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.976508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.976473 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adfd185-9df3-44db-afda-b5c67a858d54-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.976605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.976575 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.976696 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.976671 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.977149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.977128 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.978885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.978832 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.979400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.979378 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adfd185-9df3-44db-afda-b5c67a858d54-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.989127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.989098 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:24.989457 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:24.989424 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghpbj\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-kube-api-access-ghpbj\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:25.139705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:25.139625 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:25.336285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:25.336191 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:25.367471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:25.367427 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:25.437658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:25.435013 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp"] Nov 03 20:16:25.440055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:25.439830 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:25.440055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:25.439915 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:25.440055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:25.439965 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:26.258467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:26.256239 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" event={"ID":"7adfd185-9df3-44db-afda-b5c67a858d54","Type":"ContainerStarted","Data":"c6413d1816474e9cc808d8db59ba69deda2f1a242a9cba0c22a9fe777bf8338e"} Nov 03 20:16:26.258467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:26.256302 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" event={"ID":"7adfd185-9df3-44db-afda-b5c67a858d54","Type":"ContainerStarted","Data":"8e5d29a261f9255710e840393be35d6b99b90b11eb3ebafa42b41689c85e7071"} Nov 03 20:16:26.334469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:26.332793 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podStartSLOduration=2.332771804 podStartE2EDuration="2.332771804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:26.303340557 +0000 UTC m=+889.753446626" watchObservedRunningTime="2026-11-03 20:16:26.332771804 +0000 UTC m=+889.782877851" Nov 03 20:16:26.810258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:26.810115 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:26.943595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:26.943544 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:26.972858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:26.972819 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:16:27.067655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.067546 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:27.143248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.143220 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:27.153504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.153467 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:16:27.153762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.153731 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:16:27.154154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.154128 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" podUID="bc9bbc78-f089-44f5-ab9e-3a986df305d1" containerName="payload-processing" containerID="cri-o://80f6d0bbbbd447d72bfe60a374dbaf182484d517697b0f21590e14297cc854d1" gracePeriod=25 Nov 03 20:16:27.236533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.236493 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:27.260466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.257958 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:27.358515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.354963 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:27.358515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.356806 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:16:27.358515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.357038 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:16:27.450474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.443364 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" podUID="318d5333-5221-4001-893c-fc1162d1fda4" containerName="payload-processing" containerID="cri-o://78654cb0827dc31393149f4c0e5202f1314fe4fd56547f87f1d41c52bc635a62" gracePeriod=25 Nov 03 20:16:27.510029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.509975 4711 generic.go:356] "Generic (PLEG): container finished" podID="bc9bbc78-f089-44f5-ab9e-3a986df305d1" containerID="80f6d0bbbbd447d72bfe60a374dbaf182484d517697b0f21590e14297cc854d1" exitCode=0 Nov 03 20:16:27.528771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.528693 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" event={"ID":"bc9bbc78-f089-44f5-ab9e-3a986df305d1","Type":"ContainerDied","Data":"80f6d0bbbbd447d72bfe60a374dbaf182484d517697b0f21590e14297cc854d1"} Nov 03 20:16:27.837471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.833543 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:27.977649 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.976433 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:27.977649 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:27.977129 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:16:28.092475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.069080 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:28.092475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.069820 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:16:28.300468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.299804 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:28.342467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.301117 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:16:28.342467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.320538 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:28.364587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.364537 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:16:28.519493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.369148 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:16:28.519493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.369195 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:16:28.519493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.370229 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:16:28.519493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.370871 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:28.519493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.371117 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:16:28.519493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.371516 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:28.519493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.371777 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:16:28.875530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.746802 4711 generic.go:356] "Generic (PLEG): container finished" podID="318d5333-5221-4001-893c-fc1162d1fda4" containerID="78654cb0827dc31393149f4c0e5202f1314fe4fd56547f87f1d41c52bc635a62" exitCode=0 Nov 03 20:16:28.875530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.834089 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:16:28.881959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.881918 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:16:28.902915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:28.902881 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" event={"ID":"318d5333-5221-4001-893c-fc1162d1fda4","Type":"ContainerDied","Data":"78654cb0827dc31393149f4c0e5202f1314fe4fd56547f87f1d41c52bc635a62"} Nov 03 20:16:29.024535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.016056 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:16:29.081470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.073569 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:29.168610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.165474 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:16:29.168610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.165567 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:16:29.453521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.452261 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:29.475471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.458276 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:16:29.495904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.483006 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:16:29.495904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.494702 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:16:29.502123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.501545 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:29.844927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:29.844888 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:30.032794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.032753 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:16:30.116658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.116515 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:16:30.169668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.161820 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:16:30.169668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.161897 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:16:30.260468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.242589 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" event={"ID":"bc9bbc78-f089-44f5-ab9e-3a986df305d1","Type":"ContainerDied","Data":"476bb11b481b0f289c3ec55df210fb34e0ba6059b90d98706583a706e2836633"} Nov 03 20:16:30.260468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.242654 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="476bb11b481b0f289c3ec55df210fb34e0ba6059b90d98706583a706e2836633" Nov 03 20:16:30.297975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.295707 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" event={"ID":"318d5333-5221-4001-893c-fc1162d1fda4","Type":"ContainerDied","Data":"b7e5e35e073bf990e0b7b295b98a169aaaedcaf16a89f6b2bcc4e293a4a33c89"} Nov 03 20:16:30.297975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.295775 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7e5e35e073bf990e0b7b295b98a169aaaedcaf16a89f6b2bcc4e293a4a33c89" Nov 03 20:16:30.453150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.453068 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:16:30.453769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.453737 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:30.461501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.461429 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:30.493461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.493021 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:16:30.495927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.495239 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:30.496278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.496193 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:30.503622 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.502620 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:30.648435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.648397 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-routing-overlay-volume\") pod \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " Nov 03 20:16:30.648741 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.648702 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-kube-api-access-c8bzv\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-kube-api-access-c8bzv\") pod \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " Nov 03 20:16:30.648950 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.648934 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-provider-credentials\") pod \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " Nov 03 20:16:30.649190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.649175 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-plugins-config-volume\") pod \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\" (UID: \"bc9bbc78-f089-44f5-ab9e-3a986df305d1\") " Nov 03 20:16:30.653858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.653821 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-plugins-config-volume" pod "bc9bbc78-f089-44f5-ab9e-3a986df305d1" (UID: "bc9bbc78-f089-44f5-ab9e-3a986df305d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:30.654352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.654331 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-routing-overlay-volume" pod "bc9bbc78-f089-44f5-ab9e-3a986df305d1" (UID: "bc9bbc78-f089-44f5-ab9e-3a986df305d1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:30.692707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.674651 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:16:30.733108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.733011 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:30.834470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.822265 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-kube-api-access-m7hn5\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-kube-api-access-m7hn5\") pod \"318d5333-5221-4001-893c-fc1162d1fda4\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " Nov 03 20:16:30.834470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.822332 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-routing-overlay-volume\") pod \"318d5333-5221-4001-893c-fc1162d1fda4\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " Nov 03 20:16:30.834470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.822431 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-provider-credentials\") pod \"318d5333-5221-4001-893c-fc1162d1fda4\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " Nov 03 20:16:30.834470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.822496 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-plugins-config-volume\") pod \"318d5333-5221-4001-893c-fc1162d1fda4\" (UID: \"318d5333-5221-4001-893c-fc1162d1fda4\") " Nov 03 20:16:30.834470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.824244 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:30.834470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.824283 4711 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc9bbc78-f089-44f5-ab9e-3a986df305d1-routing-overlay-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:30.853154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.853108 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-plugins-config-volume" pod "318d5333-5221-4001-893c-fc1162d1fda4" (UID: "318d5333-5221-4001-893c-fc1162d1fda4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:30.924626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.924588 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-routing-overlay-volume" pod "318d5333-5221-4001-893c-fc1162d1fda4" (UID: "318d5333-5221-4001-893c-fc1162d1fda4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:30.974477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:30.965907 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-provider-credentials" pod "bc9bbc78-f089-44f5-ab9e-3a986df305d1" (UID: "bc9bbc78-f089-44f5-ab9e-3a986df305d1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:31.051969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.034087 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.146468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.116980 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-kube-api-access-c8bzv" pod "bc9bbc78-f089-44f5-ab9e-3a986df305d1" (UID: "bc9bbc78-f089-44f5-ab9e-3a986df305d1"). InnerVolumeSpecName "kube-api-access-c8bzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:31.150869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.147414 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-kube-api-access-m7hn5" pod "318d5333-5221-4001-893c-fc1162d1fda4" (UID: "318d5333-5221-4001-893c-fc1162d1fda4"). InnerVolumeSpecName "kube-api-access-m7hn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:31.150869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.147699 4711 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-routing-overlay-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:31.150869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.150709 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/318d5333-5221-4001-893c-fc1162d1fda4-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.161825 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.161876 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.163059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.165478 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-provider-credentials" pod "318d5333-5221-4001-893c-fc1162d1fda4" (UID: "318d5333-5221-4001-893c-fc1162d1fda4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.166092 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.166141 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.166174 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.166208 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.166238 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:16:31.178987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.166269 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:16:31.201855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.198877 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht"] Nov 03 20:16:31.201855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.199779 4711 cpu_manager.go:414] "containerMap: removing container" podUID="bc9bbc78-f089-44f5-ab9e-3a986df305d1" containerName="payload-processing" Nov 03 20:16:31.201855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.199798 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9bbc78-f089-44f5-ab9e-3a986df305d1" containerName="payload-processing" Nov 03 20:16:31.201855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.199825 4711 cpu_manager.go:414] "containerMap: removing container" podUID="318d5333-5221-4001-893c-fc1162d1fda4" containerName="payload-processing" Nov 03 20:16:31.201855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.199833 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="318d5333-5221-4001-893c-fc1162d1fda4" containerName="payload-processing" Nov 03 20:16:31.201855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.200033 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9bbc78-f089-44f5-ab9e-3a986df305d1" containerName="payload-processing" Nov 03 20:16:31.201855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.200049 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="318d5333-5221-4001-893c-fc1162d1fda4" containerName="payload-processing" Nov 03 20:16:31.214077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.212417 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:16:31.214077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.212467 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht"] Nov 03 20:16:31.214077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.213335 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.220620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.220385 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-085566\"/\"distinct-w2-085566-kserve-self-signed-certs\"" Nov 03 20:16:31.251621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.251590 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bzv\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-kube-api-access-c8bzv\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:31.251621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.251620 4711 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc9bbc78-f089-44f5-ab9e-3a986df305d1-provider-credentials\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:31.251815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.251636 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7hn5\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-kube-api-access-m7hn5\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:31.251815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.251648 4711 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/318d5333-5221-4001-893c-fc1162d1fda4-provider-credentials\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:31.314264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.314177 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j" Nov 03 20:16:31.315605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.315585 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6" Nov 03 20:16:31.352273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.352237 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-tmp-dir\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.352273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.352276 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-model-cache\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.352468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.352336 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f1fee8-6816-47ac-98e3-073ff518cdca-tls-certs\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.352514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.352476 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-home\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.352569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.352556 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxdx\" (UniqueName: \"kubernetes.io/projected/f2f1fee8-6816-47ac-98e3-073ff518cdca-kube-api-access-vwxdx\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.352638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.352628 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-dshm\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.363263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.363235 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j"] Nov 03 20:16:31.396482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.373303 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5cfccf547d-qz86j"] Nov 03 20:16:31.422464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.422423 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6"] Nov 03 20:16:31.428739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.428690 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-9dbdc456b-mjdh6"] Nov 03 20:16:31.454332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.454298 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-dshm\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.454802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.454373 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-tmp-dir\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.454802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.454524 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-model-cache\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.454802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.454703 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f1fee8-6816-47ac-98e3-073ff518cdca-tls-certs\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.454978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.454828 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-home\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.454978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.454898 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxdx\" (UniqueName: \"kubernetes.io/projected/f2f1fee8-6816-47ac-98e3-073ff518cdca-kube-api-access-vwxdx\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.454978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.454826 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-tmp-dir\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.455084 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.455008 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-model-cache\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.455213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.455183 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-home\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.457011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.456980 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-dshm\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.457477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.457427 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f1fee8-6816-47ac-98e3-073ff518cdca-tls-certs\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.500511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.499553 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxdx\" (UniqueName: \"kubernetes.io/projected/f2f1fee8-6816-47ac-98e3-073ff518cdca-kube-api-access-vwxdx\") pod \"distinct-w2-085566-kserve-69df6b575c-x87ht\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.532662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.532629 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:31.843105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:31.843080 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht"] Nov 03 20:16:31.844326 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:16:31.844301 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2f1fee8_6816_47ac_98e3_073ff518cdca.slice/crio-f292afe06c4d965bfeb3f6cd5f8f79d68793aedd31572aa881789808ef7cbbd0 WatchSource:0}: Error finding container f292afe06c4d965bfeb3f6cd5f8f79d68793aedd31572aa881789808ef7cbbd0: Status 404 returned error can't find the container with id f292afe06c4d965bfeb3f6cd5f8f79d68793aedd31572aa881789808ef7cbbd0 Nov 03 20:16:32.140828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:32.140786 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:16:32.140979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:32.140846 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:16:32.321675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:32.321641 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" event={"ID":"f2f1fee8-6816-47ac-98e3-073ff518cdca","Type":"ContainerStarted","Data":"f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7"} Nov 03 20:16:32.321675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:32.321681 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" event={"ID":"f2f1fee8-6816-47ac-98e3-073ff518cdca","Type":"ContainerStarted","Data":"f292afe06c4d965bfeb3f6cd5f8f79d68793aedd31572aa881789808ef7cbbd0"} Nov 03 20:16:32.343045 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:32.343003 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podStartSLOduration=2.342989409 podStartE2EDuration="2.342989409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:32.341558155 +0000 UTC m=+895.791664201" watchObservedRunningTime="2026-11-03 20:16:32.342989409 +0000 UTC m=+895.793095454" Nov 03 20:16:32.533009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:32.532953 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:32.700676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:32.700605 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318d5333-5221-4001-893c-fc1162d1fda4" path="/var/lib/kubelet/pods/318d5333-5221-4001-893c-fc1162d1fda4/volumes" Nov 03 20:16:32.701016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:32.701001 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9bbc78-f089-44f5-ab9e-3a986df305d1" path="/var/lib/kubelet/pods/bc9bbc78-f089-44f5-ab9e-3a986df305d1/volumes" Nov 03 20:16:33.042910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:33.042875 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:33.140490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:33.140434 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:16:33.140673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:33.140517 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:16:34.047648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:34.047609 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:34.140243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:34.140204 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 20:16:34.140417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:34.140276 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 20:16:35.099076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:35.098997 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:35.226580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:35.226535 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:35.226998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:35.226978 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:35.227866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:35.227842 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 20:16:36.082747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:36.082705 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:36.782942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:36.782910 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:16:36.783630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:36.783613 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:16:36.785821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:36.785801 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:16:36.786305 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:36.786290 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:16:36.917975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:36.917931 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:37.830485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:37.830407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:38.516669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:38.516634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:39.213573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:39.213527 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:40.012920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:40.012886 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:40.916043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:40.916005 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:41.532956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:41.532923 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:41.545171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:41.545142 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:42.000827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:42.000710 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:42.402184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:42.402156 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:42.807222 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:42.807191 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:43.573636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:43.573599 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:44.302113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:44.302085 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:44.964601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:44.964566 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:45.638588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:45.638557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:46.320997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:46.320914 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:46.999833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:46.999796 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:47.264848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:47.264816 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:47.532011 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:16:47.531927 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:16:47.975139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:47.975052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:49.727482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.712261 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6"] Nov 03 20:16:49.739356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.739327 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:49.809501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.795813 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6"] Nov 03 20:16:49.854086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.854055 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b13a14-openshift-default-dockercfg-r6zzp\"" Nov 03 20:16:49.967470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.947425 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glcz\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-kube-api-access-2glcz\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:49.967935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.967899 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:49.968186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.968169 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:49.968363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.968348 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:49.968577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.968560 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:49.968871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.968852 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:49.969085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.969069 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:49.969238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.969222 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:49.969379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:49.969364 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.122470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.119149 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.128482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.122818 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.128825 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.128799 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.153675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.153639 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.155549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.155512 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glcz\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-kube-api-access-2glcz\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.183785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.139665 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.208546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.153527 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:16:50.209153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.131853 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.209740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.154178 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.209867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.126591 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.209984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.183826 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.244518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.244488 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.244836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.244816 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.245012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.244995 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.253471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.245573 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.253471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.242128 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glcz\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-kube-api-access-2glcz\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.253471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.242504 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.291466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.261568 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:16:50.297148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.297109 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.298782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.298738 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:50.340300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.340264 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.401184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.400997 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.440901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.439699 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:50.475518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.473531 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.475518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.474625 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.506194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.506150 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.538470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.518710 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:50.708527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:50.703386 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:51.266389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.257641 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:16:51.266389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.258136 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Nov 03 20:16:51.266389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.258672 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:16:51.266389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.263572 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:51.266389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.264757 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:51.266389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.264884 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:51.309795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.307552 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:51.309795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.307640 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:51.545754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.509723 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:51.545754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.510114 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:51.545754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.510522 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:16:51.545754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.510852 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:16:51.545754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.527799 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:51.545754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.529402 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:16:51.545754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.529739 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:16:51.545754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.529798 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:16:51.572502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.551403 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:16:51.572502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.551738 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:16:51.718532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.716674 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:51.966564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:51.958896 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Nov 03 20:16:52.023468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.023265 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:16:52.313510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.259553 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:52.313510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.271588 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:16:52.313510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.289323 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:16:52.313510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.291736 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:16:52.540503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.536267 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:16:52.540503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.538362 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:16:52.540503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.538600 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:16:52.540503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.538784 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:16:52.540503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.540059 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:52.540503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.540142 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:52.540503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.540183 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:52.558174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.556278 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:52.558174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.556942 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded" Nov 03 20:16:52.966733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:52.966680 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.185356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.185309 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:16:53.185661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.185628 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:16:53.185851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.185835 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:16:53.186027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.186002 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:16:53.195800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.195773 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" event={"ID":"5d0a6f21-836d-4ba3-b99c-025e142e4c01","Type":"ContainerStarted","Data":"a55c6359bb13af6af8e93859013da44411986260145402c47a59c32bc4c725db"} Nov 03 20:16:53.195995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.195979 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:16:53.296470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.276957 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:53.296470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.290540 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:16:53.351567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.277559 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6"] Nov 03 20:16:53.395467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.395346 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:16:53.405469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.397153 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:16:53.481517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.480669 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:16:53.617313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:53.617230 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:54.206637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:54.206602 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" event={"ID":"5d0a6f21-836d-4ba3-b99c-025e142e4c01","Type":"ContainerStarted","Data":"fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83"} Nov 03 20:16:54.234673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:54.234621 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" podStartSLOduration=5.234606041 podStartE2EDuration="5.234606041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:54.232016008 +0000 UTC m=+917.682122054" watchObservedRunningTime="2026-11-03 20:16:54.234606041 +0000 UTC m=+917.684712086" Nov 03 20:16:54.440224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:54.440183 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:16:54.441837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:54.441809 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:16:54.442007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:54.441857 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:16:55.309588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:55.309556 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:55.440902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:55.440866 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:16:55.441293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:55.440919 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:16:55.732345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:55.732255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:56.133606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:56.133551 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:56.440850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:56.440769 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:16:56.440850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:56.440822 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:16:57.011821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:57.011783 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:57.441525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:57.440849 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:16:57.441525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:57.440942 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:16:57.758211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:57.758174 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:58.440596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:58.440559 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:16:58.440802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:58.440623 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:16:58.754238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:58.754129 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:16:59.440697 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:59.440657 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:16:59.440890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:59.440748 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:16:59.524288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:16:59.524252 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:00.341720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:00.341668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:00.480373 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:00.480327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:00.689594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:00.689514 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:17:00.690402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:00.690375 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:17:00.691230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:00.691209 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:17:01.210361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.210324 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:01.372297 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.372255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:01.724805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.720942 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl"] Nov 03 20:17:01.726406 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.726382 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.733800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.733775 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-db5557-openshift-default-dockercfg-5bpcn\"" Nov 03 20:17:01.738203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.738178 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl"] Nov 03 20:17:01.790942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.790900 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-data\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.791096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.791072 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-credential-socket\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.791149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.791097 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86609111-8d0d-4973-b295-1ac6bb633307-istio-podinfo\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.791229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.791207 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtsd\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-kube-api-access-gbtsd\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.791408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.791389 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-socket\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.791485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.791472 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-istio-token\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.791541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.791522 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86609111-8d0d-4973-b295-1ac6bb633307-istiod-ca-cert\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.791608 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.791589 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-certs\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.791642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.791620 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-envoy\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.799733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.799706 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:01.892611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.892582 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-credential-socket\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.892611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.892615 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86609111-8d0d-4973-b295-1ac6bb633307-istio-podinfo\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.892848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.892650 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtsd\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-kube-api-access-gbtsd\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.892848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.892758 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-socket\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.892848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.892817 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-istio-token\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.892848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.892831 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86609111-8d0d-4973-b295-1ac6bb633307-istiod-ca-cert\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.893070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.892859 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-certs\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.893070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.892881 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-envoy\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.893070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.892920 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-data\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.893070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.893027 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-credential-socket\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.893268 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.893197 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-socket\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.893268 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.893242 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-certs\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.893404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.893384 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-data\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.893673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.893656 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86609111-8d0d-4973-b295-1ac6bb633307-istiod-ca-cert\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.895168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.895141 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-envoy\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.895274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.895177 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86609111-8d0d-4973-b295-1ac6bb633307-istio-podinfo\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.904920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.904898 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-istio-token\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.905006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.904919 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtsd\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-kube-api-access-gbtsd\") pod \"e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:01.929590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:01.929365 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:02.043110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:02.043082 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:02.065478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:02.065410 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:02.332242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:02.332211 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl"] Nov 03 20:17:02.333614 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:17:02.333584 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86609111_8d0d_4973_b295_1ac6bb633307.slice/crio-0b4864693d9cc1c462ab7ca1e2916a89e4d196204bc4ad1baef2abe82ed5386d WatchSource:0}: Error finding container 0b4864693d9cc1c462ab7ca1e2916a89e4d196204bc4ad1baef2abe82ed5386d: Status 404 returned error can't find the container with id 0b4864693d9cc1c462ab7ca1e2916a89e4d196204bc4ad1baef2abe82ed5386d Nov 03 20:17:02.336400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:02.336369 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:02.336492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:02.336476 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:02.336535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:02.336522 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:02.907910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:02.907878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:03.281098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:03.281058 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" event={"ID":"86609111-8d0d-4973-b295-1ac6bb633307","Type":"ContainerStarted","Data":"0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d"} Nov 03 20:17:03.281284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:03.281160 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" event={"ID":"86609111-8d0d-4973-b295-1ac6bb633307","Type":"ContainerStarted","Data":"0b4864693d9cc1c462ab7ca1e2916a89e4d196204bc4ad1baef2abe82ed5386d"} Nov 03 20:17:03.310211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:03.310163 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" podStartSLOduration=2.31015086 podStartE2EDuration="2.31015086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:03.309677267 +0000 UTC m=+926.759783325" watchObservedRunningTime="2026-11-03 20:17:03.31015086 +0000 UTC m=+926.760256905" Nov 03 20:17:03.681798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:03.681711 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:04.043952 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:04.043920 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:04.049207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:04.049182 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:04.288357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:04.288325 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:04.289116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:04.289097 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:04.558402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:04.558357 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:05.300227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:05.300186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:05.513999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:05.513967 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:05.658304 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:05.658219 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:06.338691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.338653 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:06.493490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.493021 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:06.627409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.627311 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:06.809591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.809554 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:06.836995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.836966 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl"] Nov 03 20:17:06.841374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.841343 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:06.850539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.850512 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-db5557\"" Nov 03 20:17:06.850747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.850595 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-db5557-dockercfg-l4h4b\"" Nov 03 20:17:06.853062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.853038 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl"] Nov 03 20:17:06.963276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.963160 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:07.003465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:06.999408 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pt2\" (UniqueName: \"kubernetes.io/projected/0b0f1d39-99be-4ba7-b841-f8c7f941189e-kube-api-access-66pt2\") pod \"payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl\" (UID: \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:07.015488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.003923 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0f1d39-99be-4ba7-b841-f8c7f941189e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl\" (UID: \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:07.116643 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.116602 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0f1d39-99be-4ba7-b841-f8c7f941189e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl\" (UID: \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:07.116858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.116711 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pt2\" (UniqueName: \"kubernetes.io/projected/0b0f1d39-99be-4ba7-b841-f8c7f941189e-kube-api-access-66pt2\") pod \"payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl\" (UID: \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:07.117405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.117381 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0f1d39-99be-4ba7-b841-f8c7f941189e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl\" (UID: \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:07.130558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.130530 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pt2\" (UniqueName: \"kubernetes.io/projected/0b0f1d39-99be-4ba7-b841-f8c7f941189e-kube-api-access-66pt2\") pod \"payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl\" (UID: \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:07.159391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.159366 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:07.439439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.439410 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl"] Nov 03 20:17:07.440941 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:17:07.440915 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0f1d39_99be_4ba7_b841_f8c7f941189e.slice/crio-438bebb09308ca519bde4fb6c3097b4f637a901b7b4fc8ca91b7f5b1fa1b1875 WatchSource:0}: Error finding container 438bebb09308ca519bde4fb6c3097b4f637a901b7b4fc8ca91b7f5b1fa1b1875: Status 404 returned error can't find the container with id 438bebb09308ca519bde4fb6c3097b4f637a901b7b4fc8ca91b7f5b1fa1b1875 Nov 03 20:17:07.517502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.517477 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:07.669515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.667040 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:07.804913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:07.804876 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:08.315779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:08.315747 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" event={"ID":"0b0f1d39-99be-4ba7-b841-f8c7f941189e","Type":"ContainerStarted","Data":"3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a"} Nov 03 20:17:08.315947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:08.315795 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" event={"ID":"0b0f1d39-99be-4ba7-b841-f8c7f941189e","Type":"ContainerStarted","Data":"438bebb09308ca519bde4fb6c3097b4f637a901b7b4fc8ca91b7f5b1fa1b1875"} Nov 03 20:17:08.315947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:08.315935 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:08.339738 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:08.339686 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" podStartSLOduration=2.339672562 podStartE2EDuration="2.339672562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:08.33678359 +0000 UTC m=+931.786889637" watchObservedRunningTime="2026-11-03 20:17:08.339672562 +0000 UTC m=+931.789778607" Nov 03 20:17:08.475128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:08.475093 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:09.319498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:09.316701 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:11.402472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:11.398546 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:11.564470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:11.563356 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:17:11.572469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:11.569627 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:11.810607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:11.810554 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:11.968473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:11.948898 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:11.980493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:11.977541 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:12.052773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.023497 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc"] Nov 03 20:17:12.075542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.060706 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:12.102413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.102369 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:12.179525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.178553 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.203475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.201078 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-db5557-serving-cert\"" Nov 03 20:17:12.309573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.309521 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:17:12.354006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.320748 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:17:12.378534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.322306 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:17:12.380292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.359222 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9s6\" (UniqueName: \"kubernetes.io/projected/57eb2e87-84bf-464f-93fe-16ec6e168ac1-kube-api-access-7m9s6\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.380292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.380022 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.380292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.380063 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eb2e87-84bf-464f-93fe-16ec6e168ac1-openshift-service-ca\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.380292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.380102 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-metrics-certs\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.392603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.392502 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc"] Nov 03 20:17:12.435897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.414541 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:12.435897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.415299 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:17:12.483029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.469765 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:12.505497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.488315 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9s6\" (UniqueName: \"kubernetes.io/projected/57eb2e87-84bf-464f-93fe-16ec6e168ac1-kube-api-access-7m9s6\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.505497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.488553 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.505497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.488585 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eb2e87-84bf-464f-93fe-16ec6e168ac1-openshift-service-ca\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.505497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.488620 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-metrics-certs\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.505497 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:17:12.490847 4711 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-serving-cert: secret "maas-api-e2e-api-a-db5557-serving-cert" not found Nov 03 20:17:12.505497 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:17:12.490924 4711 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls podName:57eb2e87-84bf-464f-93fe-16ec6e168ac1 nodeName:}" failed. No retries permitted until 2026-11-03 20:17:12.990901597 +0000 UTC m=+936.441007640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls") pod "maas-api-e2e-api-a-db5557-7489c5556f-szjmc" (UID: "57eb2e87-84bf-464f-93fe-16ec6e168ac1") : secret "maas-api-e2e-api-a-db5557-serving-cert" not found Nov 03 20:17:12.628783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.543434 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:12.628783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.573005 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:17:12.628783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.580583 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:12.628783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.581239 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:17:12.628783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.581300 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:17:12.656075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.655041 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eb2e87-84bf-464f-93fe-16ec6e168ac1-openshift-service-ca\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.656075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.656003 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-metrics-certs\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.730517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.712110 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:12.787300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.758153 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9s6\" (UniqueName: \"kubernetes.io/projected/57eb2e87-84bf-464f-93fe-16ec6e168ac1-kube-api-access-7m9s6\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:12.859473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:12.833994 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:17:13.027663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.014432 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:13.027663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.015403 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:17:13.075699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.048352 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:17:13.076016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.075992 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:17:13.077135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.069049 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:13.077324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.069106 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:17:13.077705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.050489 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls\") pod \"maas-api-e2e-api-a-db5557-7489c5556f-szjmc\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:13.078247 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.078228 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:17:13.379529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.376282 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:17:13.379529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.376695 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:17:13.385750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.385704 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:17:13.471172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.405547 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:13.472434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.432235 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:13.477473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.471478 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:17:13.477473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.475237 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:17:13.505382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.505334 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:13.770287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.770240 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:13.827852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.775531 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:13.958212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:13.958113 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:17:14.143474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:14.137604 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:17:14.235148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:14.235036 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:14.290524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:14.274404 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:14.290524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:14.279944 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:17:14.454754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:14.454711 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded" Nov 03 20:17:14.610478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:14.458817 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:14.792469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:14.513089 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:14.792469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:14.515811 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:17:14.963535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:14.963309 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:15.135471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.093549 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:15.165493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.165429 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:15.278492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.264879 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:17:15.278492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.265015 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:17:15.288117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.288055 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:15.556198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.556170 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:17:15.557368 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.557328 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:17:15.558231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.558209 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:17:15.567606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.560678 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:17:15.721471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.720752 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:17:15.721471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.721346 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:17:15.929147 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:15.928845 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:16.017547 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:16.015241 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc"] Nov 03 20:17:16.043842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:16.043811 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:17:16.792970 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:16.792812 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" event={"ID":"57eb2e87-84bf-464f-93fe-16ec6e168ac1","Type":"ContainerStarted","Data":"95e27574ec555672897a670cf8a446acbafe19adc87a14b22af93f3f775622bb"} Nov 03 20:17:16.793617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:16.792947 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:16.793617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:16.793061 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" event={"ID":"57eb2e87-84bf-464f-93fe-16ec6e168ac1","Type":"ContainerStarted","Data":"2ae58bc4ec45e75c7eae21ff003637630d10c545981a8c84ce522a0ef01bdb31"} Nov 03 20:17:16.829790 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:16.829733 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" podStartSLOduration=5.336493215 podStartE2EDuration="5.829719235s" totalImagesPullingTime="493.22602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:16.828087347 +0000 UTC m=+940.278193395" watchObservedRunningTime="2026-11-03 20:17:16.829719235 +0000 UTC m=+940.279825279" Nov 03 20:17:17.346371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:17.346327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:17.537718 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:17:17.537682 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:17:18.655304 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:18.655262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:19.327666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:19.327632 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:17:19.357842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:19.357799 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:19.774440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:19.774388 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:20.546896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:20.546858 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:21.402137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:21.402079 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:22.133300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:22.133266 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:22.807031 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:22.807000 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:17:22.910133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:22.910097 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:23.235194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:23.235109 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:23.534780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:23.534746 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:23.723387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:23.723353 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:24.609357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:24.609319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:25.341595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:25.341561 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:26.030033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:26.029991 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:26.179692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:26.179645 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:26.819959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:26.819919 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:27.653234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:27.653192 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:28.507454 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:28.507412 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:29.203871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:29.203833 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:29.912571 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:29.912475 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:30.664836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:30.664802 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:30.792197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:30.792163 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:31.321404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.321371 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:31.468938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.468898 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:31.695643 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.695557 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk"] Nov 03 20:17:31.700264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.700240 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.713009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.712979 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk"] Nov 03 20:17:31.717818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.717784 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-18ba46-openshift-default-dockercfg-zvvnj\"" Nov 03 20:17:31.769663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.769633 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:31.799972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.799936 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.800163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.800088 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.800163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.800153 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.800284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.800248 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.800355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.800335 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.800390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.800371 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.800534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.800518 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.800618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.800550 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tjf4\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-kube-api-access-8tjf4\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.800618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.800602 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.899017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.898983 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:31.901755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.901730 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.901897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.901775 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.901897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.901858 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.901902 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.901928 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.902001 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.902030 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tjf4\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-kube-api-access-8tjf4\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.902082 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.902162 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.902239 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.902284 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.902341 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.902647 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.902911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.902892 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.904224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.904202 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.904756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.904736 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.914130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.914102 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:31.914241 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:31.914227 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tjf4\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-kube-api-access-8tjf4\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:32.016293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:32.016024 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:32.424618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:32.424589 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk"] Nov 03 20:17:32.425265 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:17:32.425233 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75b932a_f2a3_43e1_a25a_6efdf2da8740.slice/crio-0695d7d92af2a264469d994f5e910d4054fa318130c71589f4e307ada9634139 WatchSource:0}: Error finding container 0695d7d92af2a264469d994f5e910d4054fa318130c71589f4e307ada9634139: Status 404 returned error can't find the container with id 0695d7d92af2a264469d994f5e910d4054fa318130c71589f4e307ada9634139 Nov 03 20:17:32.428347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:32.428279 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:32.428579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:32.428365 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:32.428579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:32.428404 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:32.627909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:32.627870 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:32.939472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:32.928435 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" event={"ID":"c75b932a-f2a3-43e1-a25a-6efdf2da8740","Type":"ContainerStarted","Data":"7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2"} Nov 03 20:17:32.939472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:32.928517 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" event={"ID":"c75b932a-f2a3-43e1-a25a-6efdf2da8740","Type":"ContainerStarted","Data":"0695d7d92af2a264469d994f5e910d4054fa318130c71589f4e307ada9634139"} Nov 03 20:17:32.988472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:32.967123 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" podStartSLOduration=1.967103061 podStartE2EDuration="1.967103061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:32.96098067 +0000 UTC m=+956.411086728" watchObservedRunningTime="2026-11-03 20:17:32.967103061 +0000 UTC m=+956.417209107" Nov 03 20:17:33.024467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:33.019750 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:33.106499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:33.105748 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:33.949469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:33.949085 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:33.955925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:33.955662 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:17:34.759469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:34.758684 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:34.810468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:34.809540 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:17:34.853353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:34.834283 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:34.901482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:34.898668 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd"] Nov 03 20:17:34.939592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:34.939556 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:34.956755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:34.956722 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:17:35.124531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.124242 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.168546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.144607 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd"] Nov 03 20:17:35.187551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.177940 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7"] Nov 03 20:17:35.238740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.238711 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7"] Nov 03 20:17:35.239024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.239011 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.266042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.245258 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:35.329549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.303051 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.329549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.303112 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.329549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.303223 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.329549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.303264 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.329549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.303314 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.329549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.303514 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.329549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.303645 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.329549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.303718 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9sm\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-kube-api-access-4t9sm\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.329549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.303818 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.330127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.329827 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:17:35.405779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.405668 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.406104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.406081 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.406338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.406317 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.406530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.406498 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.421170 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.409234 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.421566 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.422237 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.422277 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.422347 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.422388 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.422801 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.423094 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.423150 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.425470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.424669 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:35.429471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.426326 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.439479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.422440 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.439479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.438665 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.439479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.438711 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.439479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.438797 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.439479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.438846 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlrt\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-kube-api-access-drlrt\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.439479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.438881 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.439479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.438937 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.439479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.438980 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9sm\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-kube-api-access-4t9sm\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.440049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.439873 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.488474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.481165 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9sm\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-kube-api-access-4t9sm\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.489279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.489249 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.495086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.495055 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.543471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.543382 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.543471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.543471 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.543732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.543532 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.543732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.543579 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.543732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.543680 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.543898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.543765 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.543898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.543810 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlrt\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-kube-api-access-drlrt\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.543898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.543871 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.544068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.543927 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.548530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.545335 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.548530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.545706 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-data\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.548530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.546044 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-certs\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.548530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.546590 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istiod-ca-cert\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.548530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.547210 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-credential-socket\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.548530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.547429 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:35.584475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.565743 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-envoy\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.584475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.575010 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-podinfo\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.621067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.603090 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-token\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.621067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.612694 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlrt\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-kube-api-access-drlrt\") pod \"e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.677694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.677604 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:35.783474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.780994 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:35.783474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.781804 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:17:35.812522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.812411 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:35.838470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.835325 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:17:35.840377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.838905 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:17:35.914470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:35.855115 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:36.255886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:36.192118 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:17:36.332491 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:36.272311 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:36.332491 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:36.276578 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:17:36.335052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:36.334459 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:17:36.335052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:36.334729 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Nov 03 20:17:36.461655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:36.461315 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:17:36.500783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:36.497677 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:36.691654 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:36.690362 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:17:37.347607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.347556 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:17:37.460470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.459408 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:37.556157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.554982 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:37.556157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.555136 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:37.556157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.555243 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:37.613150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.613099 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:37.613434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.613411 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:37.616633 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.616573 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:37.639918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.637917 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd"] Nov 03 20:17:37.639918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.637986 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7"] Nov 03 20:17:37.820309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.817500 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" event={"ID":"05ed1ab0-b90d-4e18-a33c-40e7da0ce032","Type":"ContainerStarted","Data":"64b1ddb640ced9e31662c8fc9a52b4db6f7ac929a176cc4d3f02cea54186aaba"} Nov 03 20:17:37.820309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.817554 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:37.821027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.820987 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" event={"ID":"2e364d3d-f437-4bd6-b0c0-8963f3cc065f","Type":"ContainerStarted","Data":"08d6a036bbab77bfc90016a5cd49712817a8ccd5e481cd8b37505730b10e8eea"} Nov 03 20:17:37.965398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:37.964649 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:38.354817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:38.354776 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:38.851464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:38.851302 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" event={"ID":"05ed1ab0-b90d-4e18-a33c-40e7da0ce032","Type":"ContainerStarted","Data":"fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11"} Nov 03 20:17:38.855046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:38.853177 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" event={"ID":"2e364d3d-f437-4bd6-b0c0-8963f3cc065f","Type":"ContainerStarted","Data":"f6f375ca6b2a4c221f991dc71bd75e6a985c782de1bc61f732df094b42808a58"} Nov 03 20:17:38.913470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:38.901233 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" podStartSLOduration=4.901208067 podStartE2EDuration="4.901208067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:38.885276735 +0000 UTC m=+962.335382795" watchObservedRunningTime="2026-11-03 20:17:38.901208067 +0000 UTC m=+962.351314115" Nov 03 20:17:38.938497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:38.936495 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" podStartSLOduration=4.936467337 podStartE2EDuration="4.936467337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:38.914137625 +0000 UTC m=+962.364243698" watchObservedRunningTime="2026-11-03 20:17:38.936467337 +0000 UTC m=+962.386573386" Nov 03 20:17:39.344737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:39.344679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:39.547882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:39.547848 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:39.549327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:39.549295 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:17:39.549440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:39.549350 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:17:39.686286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:39.683807 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:39.686911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:39.686572 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:39.686911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:39.686631 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:17:40.322466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:40.322128 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:40.547791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:40.547760 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:17:40.547962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:40.547813 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:17:40.682094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:40.682001 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:40.682719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:40.682319 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:17:41.042796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:41.042761 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:41.547954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:41.547921 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:17:41.548354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:41.547973 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:17:41.687469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:41.681840 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:41.687469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:41.681899 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:17:42.021353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:42.021305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:42.548684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:42.548645 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:17:42.549180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:42.548707 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:17:42.698612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:42.698576 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:42.720564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:42.719644 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:17:42.944060 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:42.943962 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:43.095473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:43.091975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:43.236978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:43.236871 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:43.632591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:43.628909 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:17:43.632591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:43.628983 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:17:43.680370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:43.679891 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:43.680370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:43.679958 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:17:43.996528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:43.994731 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:44.548629 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:44.548592 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:17:44.548786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:44.548674 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:17:44.678263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:44.678230 4711 patch_prober.go:29] interesting pod/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:17:44.678740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:44.678287 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:17:44.773072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:44.773034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:45.111000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.110955 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:45.812125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.812082 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:45.881112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.881082 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:45.881572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.881544 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:45.882336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.882315 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 20:17:45.892830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.892232 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:45.893663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.893633 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:45.897373 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.897349 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:17:45.975591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.975504 4711 generic.go:356] "Generic (PLEG): container finished" podID="86609111-8d0d-4973-b295-1ac6bb633307" containerID="0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d" exitCode=0 Nov 03 20:17:45.975847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.975811 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" event={"ID":"86609111-8d0d-4973-b295-1ac6bb633307","Type":"ContainerDied","Data":"0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d"} Nov 03 20:17:45.976287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.976267 4711 scope.go:122] "RemoveContainer" containerID="0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d" Nov 03 20:17:45.978308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.978271 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:45.978392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.978359 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:45.978439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:45.978405 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:46.602304 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:46.602262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:46.999441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:46.999349 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" event={"ID":"86609111-8d0d-4973-b295-1ac6bb633307","Type":"ContainerStarted","Data":"affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7"} Nov 03 20:17:47.043416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:47.043382 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:47.044534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:47.044509 4711 patch_prober.go:29] interesting pod/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:17:47.044665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:47.044553 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:17:47.395920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:47.395878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:47.532224 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:17:47.532188 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:17:48.043619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:48.043530 4711 patch_prober.go:29] interesting pod/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:17:48.043619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:48.043602 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:17:48.091258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:48.091221 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:48.748399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:48.748367 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:49.044223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:49.044177 4711 patch_prober.go:29] interesting pod/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:17:49.044694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:49.044241 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:17:49.291652 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:49.291610 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:49.529471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:49.523385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:50.043579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:50.043546 4711 patch_prober.go:29] interesting pod/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:17:50.043759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:50.043599 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:17:50.325697 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:50.325609 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:50.850458 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:50.850411 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:51.043537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:51.043501 4711 patch_prober.go:29] interesting pod/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:17:51.043738 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:51.043574 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:17:51.100059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:51.099996 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:51.602470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:51.601438 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:52.043428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:52.043397 4711 patch_prober.go:29] interesting pod/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 03 20:17:52.043618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:52.043471 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 03 20:17:52.156293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:52.156258 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:53.048230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:53.048195 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:53.049544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:53.049520 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:17:53.312401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:53.312256 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:54.091672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.091634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:54.220302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.220258 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:54.447591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.447508 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g"] Nov 03 20:17:54.451976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.451952 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.461831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.461806 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9b65143e-openshift-default-dockercfg-sd5gc\"" Nov 03 20:17:54.465001 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.464978 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g"] Nov 03 20:17:54.574310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.570548 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istiod-ca-cert\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.574310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.570662 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-data\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.574310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.570770 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmzph\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-kube-api-access-bmzph\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.574310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.570864 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-token\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.574310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.570906 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-envoy\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.574310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.571010 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-certs\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.574310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.571053 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-credential-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.574310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.571127 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-podinfo\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.574310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.571255 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.649477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.648745 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:54.774255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.774207 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-credential-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.774675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.774646 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-podinfo\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.775049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.775030 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.775252 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.775232 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istiod-ca-cert\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.775594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.775573 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-data\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.775855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.775837 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmzph\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-kube-api-access-bmzph\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.776062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.776043 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-token\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.776229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.776211 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-envoy\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.776500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.776482 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-certs\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.777095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.777074 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-certs\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.777224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.775264 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-credential-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.778211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.778192 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-data\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.795537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.786463 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istiod-ca-cert\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.795781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.787312 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-socket\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.807479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.807215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:54.817573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.813858 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-envoy\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.818162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.818129 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-podinfo\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.819097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.819071 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmzph\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-kube-api-access-bmzph\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:54.820196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:54.820171 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-token\") pod \"e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:55.105236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:55.105152 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:55.400568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:55.400535 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g"] Nov 03 20:17:55.401245 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:17:55.401220 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcdbf31_f878_4515_a5a2_e1b43b1c7420.slice/crio-25e1cb5a109b3b0755ad026e6160901359a08063193e6da3ceab62be04ec418e WatchSource:0}: Error finding container 25e1cb5a109b3b0755ad026e6160901359a08063193e6da3ceab62be04ec418e: Status 404 returned error can't find the container with id 25e1cb5a109b3b0755ad026e6160901359a08063193e6da3ceab62be04ec418e Nov 03 20:17:55.403864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:55.403833 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:55.403966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:55.403898 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:55.403966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:55.403926 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:55.652003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:55.651911 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:55.914352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:55.914245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:56.098656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:56.098624 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" event={"ID":"fdcdbf31-f878-4515-a5a2-e1b43b1c7420","Type":"ContainerStarted","Data":"b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d"} Nov 03 20:17:56.098833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:56.098676 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" event={"ID":"fdcdbf31-f878-4515-a5a2-e1b43b1c7420","Type":"ContainerStarted","Data":"25e1cb5a109b3b0755ad026e6160901359a08063193e6da3ceab62be04ec418e"} Nov 03 20:17:56.105889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:56.105862 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:56.111584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:56.111559 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:56.126983 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:56.126930 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" podStartSLOduration=2.126915668 podStartE2EDuration="2.126915668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:56.122937902 +0000 UTC m=+979.573043950" watchObservedRunningTime="2026-11-03 20:17:56.126915668 +0000 UTC m=+979.577021714" Nov 03 20:17:56.627148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:56.627114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:57.108526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:57.108491 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:57.109793 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:57.109762 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:17:57.605478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:57.605402 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:58.244949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:58.244912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:58.388384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:58.388348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:58.778476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:58.778424 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:59.449884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.449850 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8"] Nov 03 20:17:59.454430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.454410 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:17:59.465356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.464959 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9b65143e\"" Nov 03 20:17:59.466394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.466366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:59.471034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.470700 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9b65143e-dockercfg-7lkpz\"" Nov 03 20:17:59.472041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.472022 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8"] Nov 03 20:17:59.577693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.577657 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fdaa29-9500-4fbc-a1f0-077f4ff15733-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8\" (UID: \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:17:59.577863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.577805 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzxv\" (UniqueName: \"kubernetes.io/projected/00fdaa29-9500-4fbc-a1f0-077f4ff15733-kube-api-access-shzxv\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8\" (UID: \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:17:59.602596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.602561 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:59.690983 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.688384 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzxv\" (UniqueName: \"kubernetes.io/projected/00fdaa29-9500-4fbc-a1f0-077f4ff15733-kube-api-access-shzxv\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8\" (UID: \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:17:59.736286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.718772 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fdaa29-9500-4fbc-a1f0-077f4ff15733-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8\" (UID: \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:17:59.736286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.719942 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fdaa29-9500-4fbc-a1f0-077f4ff15733-plugins-config-volume\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8\" (UID: \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:17:59.737314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.737284 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzxv\" (UniqueName: \"kubernetes.io/projected/00fdaa29-9500-4fbc-a1f0-077f4ff15733-kube-api-access-shzxv\") pod \"payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8\" (UID: \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\") " pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:17:59.748467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.747551 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:17:59.788520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:17:59.786113 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:18:00.091029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:00.090992 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8"] Nov 03 20:18:00.091497 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:18:00.091468 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fdaa29_9500_4fbc_a1f0_077f4ff15733.slice/crio-cc7ea2dd17726cc9b0a57cb99e01f1678951743329676286498e5d1ef92c6148 WatchSource:0}: Error finding container cc7ea2dd17726cc9b0a57cb99e01f1678951743329676286498e5d1ef92c6148: Status 404 returned error can't find the container with id cc7ea2dd17726cc9b0a57cb99e01f1678951743329676286498e5d1ef92c6148 Nov 03 20:18:00.131459 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:00.131416 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" event={"ID":"00fdaa29-9500-4fbc-a1f0-077f4ff15733","Type":"ContainerStarted","Data":"cc7ea2dd17726cc9b0a57cb99e01f1678951743329676286498e5d1ef92c6148"} Nov 03 20:18:00.348933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:00.348852 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:00.531057 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:00.531016 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:01.139366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.139327 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" event={"ID":"00fdaa29-9500-4fbc-a1f0-077f4ff15733","Type":"ContainerStarted","Data":"dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c"} Nov 03 20:18:01.139563 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.139494 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:18:01.161792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.161744 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" podStartSLOduration=2.161729167 podStartE2EDuration="2.161729167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:01.158216948 +0000 UTC m=+984.608323006" watchObservedRunningTime="2026-11-03 20:18:01.161729167 +0000 UTC m=+984.611835214" Nov 03 20:18:01.308904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.308859 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:01.806692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.806662 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679"] Nov 03 20:18:01.811455 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.811418 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.829697 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.829667 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679"] Nov 03 20:18:01.853995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.853957 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.854158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.854014 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.854236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.854158 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300303d0-4e20-4329-a848-79d00b886977-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.854362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.854340 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.854524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.854499 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vj7m\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-kube-api-access-2vj7m\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.854608 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.854566 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300303d0-4e20-4329-a848-79d00b886977-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.854683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.854656 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.854748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.854705 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.854805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.854779 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.955686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.955641 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.955883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.955697 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.955883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.955759 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300303d0-4e20-4329-a848-79d00b886977-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.955883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.955868 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.955961 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vj7m\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-kube-api-access-2vj7m\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.956019 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300303d0-4e20-4329-a848-79d00b886977-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.956072 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.956114 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.956185 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.956217 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-credential-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.956234 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-data\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.956585 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-certs\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.956592 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-socket\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.956735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.956652 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300303d0-4e20-4329-a848-79d00b886977-istiod-ca-cert\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.958425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.958398 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-envoy\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.958812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.958790 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300303d0-4e20-4329-a848-79d00b886977-istio-podinfo\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.968001 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.967970 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-istio-token\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:01.968236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:01.968213 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vj7m\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-kube-api-access-2vj7m\") pod \"e2e-api-b-18ba46-openshift-default-7848df5f9-xn679\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:02.107706 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:02.107610 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:02.137120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:02.130186 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:02.438354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:02.438327 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679"] Nov 03 20:18:02.439835 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:18:02.439803 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300303d0_4e20_4329_a848_79d00b886977.slice/crio-b2899d7943db8e7b325de35ee266f8887ebbdb380e768deb62bf70169b503bd5 WatchSource:0}: Error finding container b2899d7943db8e7b325de35ee266f8887ebbdb380e768deb62bf70169b503bd5: Status 404 returned error can't find the container with id b2899d7943db8e7b325de35ee266f8887ebbdb380e768deb62bf70169b503bd5 Nov 03 20:18:02.442987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:02.442949 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:02.443089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:02.443034 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:02.443157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:02.443098 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:02.815423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:02.815387 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:03.033052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.033013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:03.158220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.158138 4711 generic.go:356] "Generic (PLEG): container finished" podID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerID="7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2" exitCode=0 Nov 03 20:18:03.158398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.158218 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" event={"ID":"c75b932a-f2a3-43e1-a25a-6efdf2da8740","Type":"ContainerDied","Data":"7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2"} Nov 03 20:18:03.158801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.158782 4711 scope.go:122] "RemoveContainer" containerID="7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2" Nov 03 20:18:03.160033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.160010 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" event={"ID":"300303d0-4e20-4329-a848-79d00b886977","Type":"ContainerStarted","Data":"c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b"} Nov 03 20:18:03.160141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.160063 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" event={"ID":"300303d0-4e20-4329-a848-79d00b886977","Type":"ContainerStarted","Data":"b2899d7943db8e7b325de35ee266f8887ebbdb380e768deb62bf70169b503bd5"} Nov 03 20:18:03.160482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.160429 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:03.160576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.160521 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:03.160576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.160552 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:03.218486 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.216997 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" podStartSLOduration=2.216977609 podStartE2EDuration="2.216977609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:03.216375128 +0000 UTC m=+986.666481178" watchObservedRunningTime="2026-11-03 20:18:03.216977609 +0000 UTC m=+986.667083658" Nov 03 20:18:03.617507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.617474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:03.949335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:03.949246 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:18:04.135619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.135584 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:04.137352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.137321 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 20:18:04.137501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.137369 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 20:18:04.222327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.219946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:04.223210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.222854 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" event={"ID":"c75b932a-f2a3-43e1-a25a-6efdf2da8740","Type":"ContainerStarted","Data":"702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a"} Nov 03 20:18:04.335372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.334562 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7"] Nov 03 20:18:04.339261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.339234 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.347619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.347589 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9b65143e-serving-cert\"" Nov 03 20:18:04.357226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.354289 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7"] Nov 03 20:18:04.431816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.425672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:04.431816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.430472 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-metrics-certs\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.431816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.430557 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-maas-api-tls\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.431816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.430680 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f1170-a9d8-4495-999c-1f60407db883-openshift-service-ca\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.431816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.430714 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwrz\" (UniqueName: \"kubernetes.io/projected/d14f1170-a9d8-4495-999c-1f60407db883-kube-api-access-5cwrz\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.555941 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.545674 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-metrics-certs\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.565176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.565134 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-maas-api-tls\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.588515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.565572 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f1170-a9d8-4495-999c-1f60407db883-openshift-service-ca\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.588852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.588820 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwrz\" (UniqueName: \"kubernetes.io/projected/d14f1170-a9d8-4495-999c-1f60407db883-kube-api-access-5cwrz\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.589107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.567917 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f1170-a9d8-4495-999c-1f60407db883-openshift-service-ca\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.589337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.557181 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-metrics-certs\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.591302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.591271 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:04.615208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.615167 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-maas-api-tls\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.626546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.619004 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwrz\" (UniqueName: \"kubernetes.io/projected/d14f1170-a9d8-4495-999c-1f60407db883-kube-api-access-5cwrz\") pod \"maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:04.689471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:04.688816 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:05.038818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:05.038774 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:18:05.045955 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:05.041609 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:05.045955 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:05.045618 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:18:05.135183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:05.135094 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 20:18:05.135395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:05.135373 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 20:18:05.258969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:05.258939 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7"] Nov 03 20:18:05.260974 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:18:05.260942 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14f1170_a9d8_4495_999c_1f60407db883.slice/crio-df3edbb0e5c24c3cdc3cfa3d45f93138042d4945b7876b3a666fcc36d006fc81 WatchSource:0}: Error finding container df3edbb0e5c24c3cdc3cfa3d45f93138042d4945b7876b3a666fcc36d006fc81: Status 404 returned error can't find the container with id df3edbb0e5c24c3cdc3cfa3d45f93138042d4945b7876b3a666fcc36d006fc81 Nov 03 20:18:05.351913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:05.351860 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:05.713469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:05.713058 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:06.017868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.017827 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:06.018632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.017898 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:18:06.135094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.135058 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 20:18:06.135623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.135120 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 20:18:06.248932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.248896 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" event={"ID":"d14f1170-a9d8-4495-999c-1f60407db883","Type":"ContainerStarted","Data":"f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72"} Nov 03 20:18:06.249108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.248983 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" event={"ID":"d14f1170-a9d8-4495-999c-1f60407db883","Type":"ContainerStarted","Data":"df3edbb0e5c24c3cdc3cfa3d45f93138042d4945b7876b3a666fcc36d006fc81"} Nov 03 20:18:06.249258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.249240 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:06.328336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.328210 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:06.329197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.329142 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" podStartSLOduration=1.955434567 podStartE2EDuration="2.329122745s" totalImagesPullingTime="373.688178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:06.298753357 +0000 UTC m=+989.748859406" watchObservedRunningTime="2026-11-03 20:18:06.329122745 +0000 UTC m=+989.779228793" Nov 03 20:18:06.498467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:06.487509 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:07.018296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:07.017772 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:07.018296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:07.017840 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:18:07.128296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:07.128254 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:07.144473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:07.137340 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 20:18:07.144473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:07.137401 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 20:18:07.958371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:07.958333 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:08.045252 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:08.045204 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:08.045471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:08.045300 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:18:08.134799 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:08.134765 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 20:18:08.135048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:08.135021 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 20:18:08.765260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:08.765218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:09.017852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:09.017757 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:18:09.017852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:09.017816 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:18:09.141006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:09.140964 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:09.141504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:09.141481 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:09.142147 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:09.142128 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:09.638841 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:09.622666 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl"] Nov 03 20:18:09.699417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:09.699383 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:09.833974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:09.833937 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:10.024162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:10.024130 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:18:10.024794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:10.024592 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:18:10.025683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:10.025662 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:18:10.211075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:10.211039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:10.356365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:10.356277 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:10.776250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:10.776212 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:11.197597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.197510 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc"] Nov 03 20:18:11.200767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.200729 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" podUID="57eb2e87-84bf-464f-93fe-16ec6e168ac1" containerName="maas-api" containerID="cri-o://95e27574ec555672897a670cf8a446acbafe19adc87a14b22af93f3f775622bb" gracePeriod=30 Nov 03 20:18:11.335989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.335951 4711 generic.go:356] "Generic (PLEG): container finished" podID="57eb2e87-84bf-464f-93fe-16ec6e168ac1" containerID="95e27574ec555672897a670cf8a446acbafe19adc87a14b22af93f3f775622bb" exitCode=0 Nov 03 20:18:11.336153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.336018 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" event={"ID":"57eb2e87-84bf-464f-93fe-16ec6e168ac1","Type":"ContainerDied","Data":"95e27574ec555672897a670cf8a446acbafe19adc87a14b22af93f3f775622bb"} Nov 03 20:18:11.417099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.416980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:11.508676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.508647 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:18:11.613010 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.612966 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57eb2e87-84bf-464f-93fe-16ec6e168ac1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eb2e87-84bf-464f-93fe-16ec6e168ac1-openshift-service-ca\") pod \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " Nov 03 20:18:11.613214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.613066 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-metrics-certs\") pod \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " Nov 03 20:18:11.613214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.613129 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls\") pod \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " Nov 03 20:18:11.613353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.613235 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57eb2e87-84bf-464f-93fe-16ec6e168ac1-kube-api-access-7m9s6\" (UniqueName: \"kubernetes.io/projected/57eb2e87-84bf-464f-93fe-16ec6e168ac1-kube-api-access-7m9s6\") pod \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\" (UID: \"57eb2e87-84bf-464f-93fe-16ec6e168ac1\") " Nov 03 20:18:11.614388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.614358 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57eb2e87-84bf-464f-93fe-16ec6e168ac1-openshift-service-ca" pod "57eb2e87-84bf-464f-93fe-16ec6e168ac1" (UID: "57eb2e87-84bf-464f-93fe-16ec6e168ac1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:11.616674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.616645 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-metrics-certs" pod "57eb2e87-84bf-464f-93fe-16ec6e168ac1" (UID: "57eb2e87-84bf-464f-93fe-16ec6e168ac1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:18:11.617047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.617020 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57eb2e87-84bf-464f-93fe-16ec6e168ac1-kube-api-access-7m9s6" pod "57eb2e87-84bf-464f-93fe-16ec6e168ac1" (UID: "57eb2e87-84bf-464f-93fe-16ec6e168ac1"). InnerVolumeSpecName "kube-api-access-7m9s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:11.617480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.617440 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls" pod "57eb2e87-84bf-464f-93fe-16ec6e168ac1" (UID: "57eb2e87-84bf-464f-93fe-16ec6e168ac1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:18:11.638646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.636621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:11.677603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.677569 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc"] Nov 03 20:18:11.678664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.678642 4711 cpu_manager.go:414] "containerMap: removing container" podUID="57eb2e87-84bf-464f-93fe-16ec6e168ac1" containerName="maas-api" Nov 03 20:18:11.678741 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.678667 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57eb2e87-84bf-464f-93fe-16ec6e168ac1" containerName="maas-api" Nov 03 20:18:11.678847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.678835 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="57eb2e87-84bf-464f-93fe-16ec6e168ac1" containerName="maas-api" Nov 03 20:18:11.682686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.682668 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:11.686597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.686492 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-98148a\"" Nov 03 20:18:11.686996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.686978 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-98148a-dockercfg-jb9j2\"" Nov 03 20:18:11.690556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.690527 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc"] Nov 03 20:18:11.715708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.715673 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff857a31-27e0-4e2b-81c5-d21795202723-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc\" (UID: \"ff857a31-27e0-4e2b-81c5-d21795202723\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:11.716138 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.716076 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bqs2\" (UniqueName: \"kubernetes.io/projected/ff857a31-27e0-4e2b-81c5-d21795202723-kube-api-access-9bqs2\") pod \"payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc\" (UID: \"ff857a31-27e0-4e2b-81c5-d21795202723\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:11.716366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.716347 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57eb2e87-84bf-464f-93fe-16ec6e168ac1-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:11.716543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.716518 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:11.716543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.716545 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57eb2e87-84bf-464f-93fe-16ec6e168ac1-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:11.716723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.716560 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9s6\" (UniqueName: \"kubernetes.io/projected/57eb2e87-84bf-464f-93fe-16ec6e168ac1-kube-api-access-7m9s6\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:11.771931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.771896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:11.822467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.821521 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff857a31-27e0-4e2b-81c5-d21795202723-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc\" (UID: \"ff857a31-27e0-4e2b-81c5-d21795202723\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:11.822467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.821761 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bqs2\" (UniqueName: \"kubernetes.io/projected/ff857a31-27e0-4e2b-81c5-d21795202723-kube-api-access-9bqs2\") pod \"payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc\" (UID: \"ff857a31-27e0-4e2b-81c5-d21795202723\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:11.834468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.823135 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff857a31-27e0-4e2b-81c5-d21795202723-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc\" (UID: \"ff857a31-27e0-4e2b-81c5-d21795202723\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:11.885469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:11.876325 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bqs2\" (UniqueName: \"kubernetes.io/projected/ff857a31-27e0-4e2b-81c5-d21795202723-kube-api-access-9bqs2\") pod \"payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc\" (UID: \"ff857a31-27e0-4e2b-81c5-d21795202723\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:12.002856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.002760 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:12.042771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.042727 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:12.154189 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.154162 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:18:12.262753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.262729 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:18:12.300484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.300435 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc"] Nov 03 20:18:12.346091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.346060 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" event={"ID":"57eb2e87-84bf-464f-93fe-16ec6e168ac1","Type":"ContainerDied","Data":"2ae58bc4ec45e75c7eae21ff003637630d10c545981a8c84ce522a0ef01bdb31"} Nov 03 20:18:12.346259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.346122 4711 scope.go:122] "RemoveContainer" containerID="95e27574ec555672897a670cf8a446acbafe19adc87a14b22af93f3f775622bb" Nov 03 20:18:12.346259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.346073 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc" Nov 03 20:18:12.347709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.347674 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" event={"ID":"ff857a31-27e0-4e2b-81c5-d21795202723","Type":"ContainerStarted","Data":"7c2809be58820c4baf1791366f982fb7abc608a266ae303137ef71d586d4f41e"} Nov 03 20:18:12.387424 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.387401 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc"] Nov 03 20:18:12.392846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.392824 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db5557-7489c5556f-szjmc"] Nov 03 20:18:12.682746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:12.682670 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57eb2e87-84bf-464f-93fe-16ec6e168ac1" path="/var/lib/kubelet/pods/57eb2e87-84bf-464f-93fe-16ec6e168ac1/volumes" Nov 03 20:18:13.193360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:13.193324 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:13.360253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:13.360148 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" event={"ID":"ff857a31-27e0-4e2b-81c5-d21795202723","Type":"ContainerStarted","Data":"90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e"} Nov 03 20:18:13.361349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:13.361325 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:13.388390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:13.388325 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" podStartSLOduration=2.388307147 podStartE2EDuration="2.388307147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:13.384417806 +0000 UTC m=+996.834523856" watchObservedRunningTime="2026-11-03 20:18:13.388307147 +0000 UTC m=+996.838413195" Nov 03 20:18:13.486349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:13.486265 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:13.964899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:13.964860 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:14.636215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:14.636170 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" podUID="0b0f1d39-99be-4ba7-b841-f8c7f941189e" containerName="payload-pre-processing" containerID="cri-o://3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a" gracePeriod=25 Nov 03 20:18:14.831281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:14.831138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:14.970245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:14.970212 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:18:15.073699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.073667 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0f1d39-99be-4ba7-b841-f8c7f941189e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0f1d39-99be-4ba7-b841-f8c7f941189e-plugins-config-volume\") pod \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\" (UID: \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\") " Nov 03 20:18:15.073887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.073760 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0f1d39-99be-4ba7-b841-f8c7f941189e-kube-api-access-66pt2\" (UniqueName: \"kubernetes.io/projected/0b0f1d39-99be-4ba7-b841-f8c7f941189e-kube-api-access-66pt2\") pod \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\" (UID: \"0b0f1d39-99be-4ba7-b841-f8c7f941189e\") " Nov 03 20:18:15.074107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.074081 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0f1d39-99be-4ba7-b841-f8c7f941189e-plugins-config-volume" pod "0b0f1d39-99be-4ba7-b841-f8c7f941189e" (UID: "0b0f1d39-99be-4ba7-b841-f8c7f941189e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:15.074332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.074314 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b0f1d39-99be-4ba7-b841-f8c7f941189e-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:15.075880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.075849 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0f1d39-99be-4ba7-b841-f8c7f941189e-kube-api-access-66pt2" pod "0b0f1d39-99be-4ba7-b841-f8c7f941189e" (UID: "0b0f1d39-99be-4ba7-b841-f8c7f941189e"). InnerVolumeSpecName "kube-api-access-66pt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:15.176034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.176001 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66pt2\" (UniqueName: \"kubernetes.io/projected/0b0f1d39-99be-4ba7-b841-f8c7f941189e-kube-api-access-66pt2\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:15.375972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.375940 4711 generic.go:356] "Generic (PLEG): container finished" podID="0b0f1d39-99be-4ba7-b841-f8c7f941189e" containerID="3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a" exitCode=0 Nov 03 20:18:15.376158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.375985 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" Nov 03 20:18:15.376158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.376024 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" event={"ID":"0b0f1d39-99be-4ba7-b841-f8c7f941189e","Type":"ContainerDied","Data":"3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a"} Nov 03 20:18:15.376158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.376080 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl" event={"ID":"0b0f1d39-99be-4ba7-b841-f8c7f941189e","Type":"ContainerDied","Data":"438bebb09308ca519bde4fb6c3097b4f637a901b7b4fc8ca91b7f5b1fa1b1875"} Nov 03 20:18:15.376158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.376103 4711 scope.go:122] "RemoveContainer" containerID="3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a" Nov 03 20:18:15.386550 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.386534 4711 scope.go:122] "RemoveContainer" containerID="3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a" Nov 03 20:18:15.386818 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:18:15.386801 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a\": container with ID starting with 3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a not found: ID does not exist" containerID="3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a" Nov 03 20:18:15.386874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.386827 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a"} err="failed to get container status \"3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a\": rpc error: code = NotFound desc = could not find container \"3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a\": container with ID starting with 3c0e19c05301cd366f8a1c8209028b82f5f9c71a279314e1443180c9726b6b1a not found: ID does not exist" Nov 03 20:18:15.401081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.401057 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl"] Nov 03 20:18:15.404424 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.404404 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db5557-66df8bbf95-fqrzl"] Nov 03 20:18:15.494565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:15.494532 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:16.135987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:16.135938 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:16.269810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:16.269774 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:16.611320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:16.611283 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:16.683705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:16.683673 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0f1d39-99be-4ba7-b841-f8c7f941189e" path="/var/lib/kubelet/pods/0b0f1d39-99be-4ba7-b841-f8c7f941189e/volumes" Nov 03 20:18:16.904863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:16.904785 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl"] Nov 03 20:18:16.905017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:16.904987 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" containerID="cri-o://affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7" gracePeriod=30 Nov 03 20:18:16.976880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:16.976844 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:17.403699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:17.403662 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:17.536112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:17.536069 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:17.548299 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:18:17.548265 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:18:17.793537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:17.793498 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:18.324205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:18.324161 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:18.466469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:18.466284 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:19.340425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:19.340382 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:20.542469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:20.542369 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:21.065470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.064097 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:21.097470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.091739 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck"] Nov 03 20:18:21.122469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.109195 4711 cpu_manager.go:414] "containerMap: removing container" podUID="0b0f1d39-99be-4ba7-b841-f8c7f941189e" containerName="payload-pre-processing" Nov 03 20:18:21.122469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.109229 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0f1d39-99be-4ba7-b841-f8c7f941189e" containerName="payload-pre-processing" Nov 03 20:18:21.122469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.109437 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0f1d39-99be-4ba7-b841-f8c7f941189e" containerName="payload-pre-processing" Nov 03 20:18:21.230332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.230294 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck"] Nov 03 20:18:21.256048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.237292 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.315525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.286759 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:18:21.336471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.334373 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kbftm\"" Nov 03 20:18:21.352483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.349762 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:21.352483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.350894 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.352483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.350978 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.352483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.351113 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.352483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.351222 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.352483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.351255 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6826f58b-3c04-4873-a9e7-970211daf43b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.352483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.351358 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.352483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.351468 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhl4q\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-kube-api-access-zhl4q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.418473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.418199 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:18:21.418652 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.418527 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:18:21.565941 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.516342 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.565941 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.560614 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.601821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.576488 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhl4q\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-kube-api-access-zhl4q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.601821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.576656 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.601821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.576743 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.601821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.576884 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.601821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.576997 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.601821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.577028 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6826f58b-3c04-4873-a9e7-970211daf43b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.624470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.620941 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.648498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.648434 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhl4q\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-kube-api-access-zhl4q\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.686469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.663065 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.686469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.679047 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.705142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.690958 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.737465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.736626 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:21.760278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.740795 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6826f58b-3c04-4873-a9e7-970211daf43b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kgcck\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:21.893849 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:21.893746 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:22.355476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.352593 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:22.456855 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:18:22.456813 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6826f58b_3c04_4873_a9e7_970211daf43b.slice/crio-33a52d8adc2b83fa3b89065f19a491acc72aa53c850f1bf82f265dd0189f3f3d WatchSource:0}: Error finding container 33a52d8adc2b83fa3b89065f19a491acc72aa53c850f1bf82f265dd0189f3f3d: Status 404 returned error can't find the container with id 33a52d8adc2b83fa3b89065f19a491acc72aa53c850f1bf82f265dd0189f3f3d Nov 03 20:18:22.464283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.462210 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck"] Nov 03 20:18:22.552990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.552962 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:18:22.640495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.640459 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-certs\") pod \"86609111-8d0d-4973-b295-1ac6bb633307\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " Nov 03 20:18:22.641004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.640564 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-credential-socket\") pod \"86609111-8d0d-4973-b295-1ac6bb633307\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " Nov 03 20:18:22.641004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.640621 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86609111-8d0d-4973-b295-1ac6bb633307-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86609111-8d0d-4973-b295-1ac6bb633307-istio-podinfo\") pod \"86609111-8d0d-4973-b295-1ac6bb633307\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " Nov 03 20:18:22.641004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.640720 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-envoy\") pod \"86609111-8d0d-4973-b295-1ac6bb633307\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " Nov 03 20:18:22.641004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.640733 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-certs" pod "86609111-8d0d-4973-b295-1ac6bb633307" (UID: "86609111-8d0d-4973-b295-1ac6bb633307"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:22.641004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.640829 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86609111-8d0d-4973-b295-1ac6bb633307-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86609111-8d0d-4973-b295-1ac6bb633307-istiod-ca-cert\") pod \"86609111-8d0d-4973-b295-1ac6bb633307\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " Nov 03 20:18:22.641004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.640850 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-credential-socket" pod "86609111-8d0d-4973-b295-1ac6bb633307" (UID: "86609111-8d0d-4973-b295-1ac6bb633307"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:22.641004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.640910 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-kube-api-access-gbtsd\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-kube-api-access-gbtsd\") pod \"86609111-8d0d-4973-b295-1ac6bb633307\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " Nov 03 20:18:22.641004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.640951 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-socket\") pod \"86609111-8d0d-4973-b295-1ac6bb633307\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " Nov 03 20:18:22.641309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.641174 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-istio-token\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-istio-token\") pod \"86609111-8d0d-4973-b295-1ac6bb633307\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " Nov 03 20:18:22.641309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.641203 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86609111-8d0d-4973-b295-1ac6bb633307-istiod-ca-cert" pod "86609111-8d0d-4973-b295-1ac6bb633307" (UID: "86609111-8d0d-4973-b295-1ac6bb633307"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:22.641309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.641271 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-data\") pod \"86609111-8d0d-4973-b295-1ac6bb633307\" (UID: \"86609111-8d0d-4973-b295-1ac6bb633307\") " Nov 03 20:18:22.641486 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.641309 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-socket" pod "86609111-8d0d-4973-b295-1ac6bb633307" (UID: "86609111-8d0d-4973-b295-1ac6bb633307"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:22.641589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.641569 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-data" pod "86609111-8d0d-4973-b295-1ac6bb633307" (UID: "86609111-8d0d-4973-b295-1ac6bb633307"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:22.641966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.641949 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:22.642050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.641970 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:22.642050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.641985 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:22.642050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.641997 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86609111-8d0d-4973-b295-1ac6bb633307-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:22.642050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.642010 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:22.643319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.643294 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86609111-8d0d-4973-b295-1ac6bb633307-istio-podinfo" pod "86609111-8d0d-4973-b295-1ac6bb633307" (UID: "86609111-8d0d-4973-b295-1ac6bb633307"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:18:22.643574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.643550 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-envoy" pod "86609111-8d0d-4973-b295-1ac6bb633307" (UID: "86609111-8d0d-4973-b295-1ac6bb633307"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:22.643696 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.643627 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-istio-token" pod "86609111-8d0d-4973-b295-1ac6bb633307" (UID: "86609111-8d0d-4973-b295-1ac6bb633307"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:22.643822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.643788 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-kube-api-access-gbtsd" pod "86609111-8d0d-4973-b295-1ac6bb633307" (UID: "86609111-8d0d-4973-b295-1ac6bb633307"). InnerVolumeSpecName "kube-api-access-gbtsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:22.713712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.713680 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" event={"ID":"6826f58b-3c04-4873-a9e7-970211daf43b","Type":"ContainerStarted","Data":"33a52d8adc2b83fa3b89065f19a491acc72aa53c850f1bf82f265dd0189f3f3d"} Nov 03 20:18:22.715348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.715321 4711 generic.go:356] "Generic (PLEG): container finished" podID="86609111-8d0d-4973-b295-1ac6bb633307" containerID="affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7" exitCode=0 Nov 03 20:18:22.715477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.715369 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" Nov 03 20:18:22.715477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.715396 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" event={"ID":"86609111-8d0d-4973-b295-1ac6bb633307","Type":"ContainerDied","Data":"affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7"} Nov 03 20:18:22.715477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.715462 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl" event={"ID":"86609111-8d0d-4973-b295-1ac6bb633307","Type":"ContainerDied","Data":"0b4864693d9cc1c462ab7ca1e2916a89e4d196204bc4ad1baef2abe82ed5386d"} Nov 03 20:18:22.715596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.715490 4711 scope.go:122] "RemoveContainer" containerID="affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7" Nov 03 20:18:22.725876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.725855 4711 scope.go:122] "RemoveContainer" containerID="0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d" Nov 03 20:18:22.736634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.736602 4711 scope.go:122] "RemoveContainer" containerID="affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7" Nov 03 20:18:22.736992 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:18:22.736969 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7\": container with ID starting with affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7 not found: ID does not exist" containerID="affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7" Nov 03 20:18:22.737083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.737001 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7"} err="failed to get container status \"affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7\": rpc error: code = NotFound desc = could not find container \"affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7\": container with ID starting with affdfefeac5732d919e61935e40a097a67f6e47b60b4bdb786bc7ec017c668a7 not found: ID does not exist" Nov 03 20:18:22.737083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.737019 4711 scope.go:122] "RemoveContainer" containerID="0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d" Nov 03 20:18:22.737367 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:18:22.737344 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d\": container with ID starting with 0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d not found: ID does not exist" containerID="0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d" Nov 03 20:18:22.737433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.737371 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d"} err="failed to get container status \"0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d\": rpc error: code = NotFound desc = could not find container \"0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d\": container with ID starting with 0af5aa869c66137049e2cff01f7a6b08af608f8cbcea3cde4fc9e36b12f0db7d not found: ID does not exist" Nov 03 20:18:22.743383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.743358 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86609111-8d0d-4973-b295-1ac6bb633307-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:22.743482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.743384 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86609111-8d0d-4973-b295-1ac6bb633307-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:22.743482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.743396 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbtsd\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-kube-api-access-gbtsd\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:22.743482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.743405 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86609111-8d0d-4973-b295-1ac6bb633307-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:22.748859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.748836 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl"] Nov 03 20:18:22.756919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:22.756896 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-db5557-openshift-default-7b85fc7bbf-w8xtl"] Nov 03 20:18:23.510557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:23.510520 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:23.902028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:23.901891 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:24.029777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:24.029742 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:24.718878 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:24.718839 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:24.721702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:24.721672 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86609111-8d0d-4973-b295-1ac6bb633307" path="/var/lib/kubelet/pods/86609111-8d0d-4973-b295-1ac6bb633307/volumes" Nov 03 20:18:25.392974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:25.392944 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:18:25.464184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:25.464150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:25.692624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:25.692480 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:26.253074 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:26.253024 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:26.887136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:26.887104 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:26.887419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:26.887166 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:27.290308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:27.290272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:27.759029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:27.758996 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" event={"ID":"6826f58b-3c04-4873-a9e7-970211daf43b","Type":"ContainerStarted","Data":"e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb"} Nov 03 20:18:27.759227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:27.759182 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:27.761000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:27.760969 4711 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kgcck container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:18:27.761137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:27.761023 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" podUID="6826f58b-3c04-4873-a9e7-970211daf43b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:18:27.791676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:27.791603 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" podStartSLOduration=3.367325411 podStartE2EDuration="7.791581473s" totalImagesPullingTime="4.424256062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:27.786669436 +0000 UTC m=+1011.236775495" watchObservedRunningTime="2026-11-03 20:18:27.791581473 +0000 UTC m=+1011.241687521" Nov 03 20:18:28.150664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:28.150566 4711 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:18:28.150664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:28.150642 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:28.314258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:28.314219 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:28.769927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:28.769895 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 20:18:29.183572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:29.183420 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:29.892779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:29.892740 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:30.235353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:30.235239 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:30.493095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:30.493007 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:30.787734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:30.787699 4711 generic.go:356] "Generic (PLEG): container finished" podID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerID="b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d" exitCode=0 Nov 03 20:18:30.787908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:30.787785 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" event={"ID":"fdcdbf31-f878-4515-a5a2-e1b43b1c7420","Type":"ContainerDied","Data":"b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d"} Nov 03 20:18:30.788330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:30.788307 4711 scope.go:122] "RemoveContainer" containerID="b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d" Nov 03 20:18:30.789994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:30.789965 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:30.790095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:30.790031 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:30.790095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:30.790067 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:30.882027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:30.881999 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:31.015998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:31.015961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:31.492398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:31.492184 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:31.631494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:31.631359 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:31.802206 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:31.802169 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" event={"ID":"fdcdbf31-f878-4515-a5a2-e1b43b1c7420","Type":"ContainerStarted","Data":"9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22"} Nov 03 20:18:32.058097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:32.057987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:32.106003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:32.105972 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:18:32.106247 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:32.106206 4711 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:18:32.106396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:32.106260 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:18:33.095214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:33.095167 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:33.105967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:33.105936 4711 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:18:33.106137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:33.105992 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:18:33.333412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:33.333360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:33.464608 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:33.464511 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:33.804051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:33.804014 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:34.106537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:34.106388 4711 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:18:34.106537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:34.106465 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:18:34.109550 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:34.109520 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679"] Nov 03 20:18:34.109930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:34.109903 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" containerID="cri-o://c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b" gracePeriod=30 Nov 03 20:18:34.112898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:34.112876 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk"] Nov 03 20:18:34.113357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:34.113296 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" containerID="cri-o://702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a" gracePeriod=30 Nov 03 20:18:34.149779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:34.149739 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:34.712400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:34.712366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:35.106187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:35.106154 4711 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:18:35.106365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:35.106208 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:18:35.715559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:35.715515 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:35.906521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:35.906488 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:36.106213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:36.106176 4711 patch_prober.go:29] interesting pod/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:18:36.106417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:36.106240 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:18:36.114657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:36.114598 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:36.967650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:36.967614 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:37.107308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:37.107272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:37.233521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:37.230615 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:18:37.233521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:37.233355 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:18:37.235969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:37.235944 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:18:37.349760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:37.349722 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:38.352757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:38.352724 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:39.143253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.143220 4711 patch_prober.go:29] interesting pod/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 20:18:39.143416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.143297 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 20:18:39.338776 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.338745 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:39.443698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.443667 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:39.495568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.495531 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:18:39.558209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558176 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-socket\") pod \"300303d0-4e20-4329-a848-79d00b886977\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " Nov 03 20:18:39.558392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558221 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-kube-api-access-8tjf4\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-kube-api-access-8tjf4\") pod \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " Nov 03 20:18:39.558392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558297 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-data\") pod \"300303d0-4e20-4329-a848-79d00b886977\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " Nov 03 20:18:39.558392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558329 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-credential-socket\") pod \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " Nov 03 20:18:39.558392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558375 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-certs\") pod \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " Nov 03 20:18:39.558678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558420 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-credential-socket\") pod \"300303d0-4e20-4329-a848-79d00b886977\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " Nov 03 20:18:39.558678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558469 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-envoy\") pod \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " Nov 03 20:18:39.558678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558520 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300303d0-4e20-4329-a848-79d00b886977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300303d0-4e20-4329-a848-79d00b886977-istiod-ca-cert\") pod \"300303d0-4e20-4329-a848-79d00b886977\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " Nov 03 20:18:39.558678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558567 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-socket\") pod \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " Nov 03 20:18:39.558678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558582 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-socket" pod "300303d0-4e20-4329-a848-79d00b886977" (UID: "300303d0-4e20-4329-a848-79d00b886977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.558678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558594 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-podinfo\") pod \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " Nov 03 20:18:39.558678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558648 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-data" pod "300303d0-4e20-4329-a848-79d00b886977" (UID: "300303d0-4e20-4329-a848-79d00b886977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.559052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558701 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-istio-token\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-istio-token\") pod \"300303d0-4e20-4329-a848-79d00b886977\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " Nov 03 20:18:39.559052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558731 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istiod-ca-cert\") pod \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " Nov 03 20:18:39.559052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.558848 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-data\") pod \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " Nov 03 20:18:39.559204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559059 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-credential-socket" pod "300303d0-4e20-4329-a848-79d00b886977" (UID: "300303d0-4e20-4329-a848-79d00b886977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.559204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559080 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300303d0-4e20-4329-a848-79d00b886977-istiod-ca-cert" pod "300303d0-4e20-4329-a848-79d00b886977" (UID: "300303d0-4e20-4329-a848-79d00b886977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:39.559204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559113 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-socket" pod "c75b932a-f2a3-43e1-a25a-6efdf2da8740" (UID: "c75b932a-f2a3-43e1-a25a-6efdf2da8740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.559362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559285 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-envoy\") pod \"300303d0-4e20-4329-a848-79d00b886977\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " Nov 03 20:18:39.559362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559329 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-token\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-token\") pod \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\" (UID: \"c75b932a-f2a3-43e1-a25a-6efdf2da8740\") " Nov 03 20:18:39.559492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559396 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-kube-api-access-2vj7m\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-kube-api-access-2vj7m\") pod \"300303d0-4e20-4329-a848-79d00b886977\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " Nov 03 20:18:39.559492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559426 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-certs\") pod \"300303d0-4e20-4329-a848-79d00b886977\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " Nov 03 20:18:39.559492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559488 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/300303d0-4e20-4329-a848-79d00b886977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300303d0-4e20-4329-a848-79d00b886977-istio-podinfo\") pod \"300303d0-4e20-4329-a848-79d00b886977\" (UID: \"300303d0-4e20-4329-a848-79d00b886977\") " Nov 03 20:18:39.559658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559549 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-data" pod "c75b932a-f2a3-43e1-a25a-6efdf2da8740" (UID: "c75b932a-f2a3-43e1-a25a-6efdf2da8740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559799 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istiod-ca-cert" pod "c75b932a-f2a3-43e1-a25a-6efdf2da8740" (UID: "c75b932a-f2a3-43e1-a25a-6efdf2da8740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.559804 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-certs" pod "300303d0-4e20-4329-a848-79d00b886977" (UID: "300303d0-4e20-4329-a848-79d00b886977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.560319 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.560342 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.560358 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.560372 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.560387 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.560400 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.560416 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300303d0-4e20-4329-a848-79d00b886977-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.560472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.560430 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.561338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.561313 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-certs" pod "c75b932a-f2a3-43e1-a25a-6efdf2da8740" (UID: "c75b932a-f2a3-43e1-a25a-6efdf2da8740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.561483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.561336 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-podinfo" pod "c75b932a-f2a3-43e1-a25a-6efdf2da8740" (UID: "c75b932a-f2a3-43e1-a25a-6efdf2da8740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:18:39.561483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.561375 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-credential-socket" pod "c75b932a-f2a3-43e1-a25a-6efdf2da8740" (UID: "c75b932a-f2a3-43e1-a25a-6efdf2da8740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.561886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.561854 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-envoy" pod "300303d0-4e20-4329-a848-79d00b886977" (UID: "300303d0-4e20-4329-a848-79d00b886977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.562954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.562927 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-kube-api-access-2vj7m" pod "300303d0-4e20-4329-a848-79d00b886977" (UID: "300303d0-4e20-4329-a848-79d00b886977"). InnerVolumeSpecName "kube-api-access-2vj7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:39.563623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.563586 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/300303d0-4e20-4329-a848-79d00b886977-istio-podinfo" pod "300303d0-4e20-4329-a848-79d00b886977" (UID: "300303d0-4e20-4329-a848-79d00b886977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:18:39.563787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.563766 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-kube-api-access-8tjf4" pod "c75b932a-f2a3-43e1-a25a-6efdf2da8740" (UID: "c75b932a-f2a3-43e1-a25a-6efdf2da8740"). InnerVolumeSpecName "kube-api-access-8tjf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:39.563881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.563824 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-envoy" pod "c75b932a-f2a3-43e1-a25a-6efdf2da8740" (UID: "c75b932a-f2a3-43e1-a25a-6efdf2da8740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:18:39.564823 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.564799 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-istio-token" pod "300303d0-4e20-4329-a848-79d00b886977" (UID: "300303d0-4e20-4329-a848-79d00b886977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:39.565879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.565856 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-token" pod "c75b932a-f2a3-43e1-a25a-6efdf2da8740" (UID: "c75b932a-f2a3-43e1-a25a-6efdf2da8740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:39.662351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662264 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tjf4\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-kube-api-access-8tjf4\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.662351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662299 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.662351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662309 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.662351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662318 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.662351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662327 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.662351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662336 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.662351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662345 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300303d0-4e20-4329-a848-79d00b886977-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.662351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662353 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75b932a-f2a3-43e1-a25a-6efdf2da8740-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.662728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662362 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vj7m\" (UniqueName: \"kubernetes.io/projected/300303d0-4e20-4329-a848-79d00b886977-kube-api-access-2vj7m\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.662728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.662371 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300303d0-4e20-4329-a848-79d00b886977-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:39.873108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.873075 4711 generic.go:356] "Generic (PLEG): container finished" podID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerID="702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a" exitCode=0 Nov 03 20:18:39.873284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.873125 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" Nov 03 20:18:39.873284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.873166 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" event={"ID":"c75b932a-f2a3-43e1-a25a-6efdf2da8740","Type":"ContainerDied","Data":"702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a"} Nov 03 20:18:39.873284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.873226 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk" event={"ID":"c75b932a-f2a3-43e1-a25a-6efdf2da8740","Type":"ContainerDied","Data":"0695d7d92af2a264469d994f5e910d4054fa318130c71589f4e307ada9634139"} Nov 03 20:18:39.873284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.873258 4711 scope.go:122] "RemoveContainer" containerID="702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a" Nov 03 20:18:39.874644 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.874624 4711 generic.go:356] "Generic (PLEG): container finished" podID="300303d0-4e20-4329-a848-79d00b886977" containerID="c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b" exitCode=0 Nov 03 20:18:39.874756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.874668 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" Nov 03 20:18:39.874756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.874716 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" event={"ID":"300303d0-4e20-4329-a848-79d00b886977","Type":"ContainerDied","Data":"c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b"} Nov 03 20:18:39.874891 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.874762 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679" event={"ID":"300303d0-4e20-4329-a848-79d00b886977","Type":"ContainerDied","Data":"b2899d7943db8e7b325de35ee266f8887ebbdb380e768deb62bf70169b503bd5"} Nov 03 20:18:39.884103 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.884085 4711 scope.go:122] "RemoveContainer" containerID="7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2" Nov 03 20:18:39.893712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.893690 4711 scope.go:122] "RemoveContainer" containerID="702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a" Nov 03 20:18:39.893982 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:18:39.893962 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a\": container with ID starting with 702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a not found: ID does not exist" containerID="702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a" Nov 03 20:18:39.894049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.893990 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a"} err="failed to get container status \"702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a\": rpc error: code = NotFound desc = could not find container \"702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a\": container with ID starting with 702f5460003f649f6f9e4831761cf0c8c84d66d66dc7e4a603f0164a274e2e6a not found: ID does not exist" Nov 03 20:18:39.894049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.894009 4711 scope.go:122] "RemoveContainer" containerID="7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2" Nov 03 20:18:39.894243 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:18:39.894227 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2\": container with ID starting with 7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2 not found: ID does not exist" containerID="7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2" Nov 03 20:18:39.894285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.894250 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2"} err="failed to get container status \"7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2\": rpc error: code = NotFound desc = could not find container \"7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2\": container with ID starting with 7ca81158d9c8bdf4d95ed2a0631edddd147307c70b1964b6197ea3138dec58b2 not found: ID does not exist" Nov 03 20:18:39.894285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.894264 4711 scope.go:122] "RemoveContainer" containerID="c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b" Nov 03 20:18:39.903038 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.903008 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679"] Nov 03 20:18:39.904507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.904488 4711 scope.go:122] "RemoveContainer" containerID="c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b" Nov 03 20:18:39.904874 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:18:39.904854 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b\": container with ID starting with c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b not found: ID does not exist" containerID="c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b" Nov 03 20:18:39.904962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.904881 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b"} err="failed to get container status \"c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b\": rpc error: code = NotFound desc = could not find container \"c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b\": container with ID starting with c75daddf22280431486d0e1927ff561841deed9bb47b4181ce339867f708477b not found: ID does not exist" Nov 03 20:18:39.907338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.907313 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-xn679"] Nov 03 20:18:39.920044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.919975 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk"] Nov 03 20:18:39.924216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:39.924186 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18ba46-openshift-default-7848df5f9-cgknk"] Nov 03 20:18:40.023691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:40.023662 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:40.682215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:40.682181 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300303d0-4e20-4329-a848-79d00b886977" path="/var/lib/kubelet/pods/300303d0-4e20-4329-a848-79d00b886977/volumes" Nov 03 20:18:40.682685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:40.682669 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" path="/var/lib/kubelet/pods/c75b932a-f2a3-43e1-a25a-6efdf2da8740/volumes" Nov 03 20:18:40.694129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:40.694101 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:41.395920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:41.395887 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:42.055911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:42.055872 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:42.723758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:42.723705 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:42.855521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:42.855486 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:42.988048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:42.987956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:43.601341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:43.601308 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:44.323071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.323034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:44.484817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.484776 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt"] Nov 03 20:18:44.485473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485459 4711 cpu_manager.go:414] "containerMap: removing container" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" Nov 03 20:18:44.485525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485474 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" Nov 03 20:18:44.485525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485485 4711 cpu_manager.go:414] "containerMap: removing container" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" Nov 03 20:18:44.485525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485490 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" Nov 03 20:18:44.485525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485516 4711 cpu_manager.go:414] "containerMap: removing container" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" Nov 03 20:18:44.485525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485521 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" Nov 03 20:18:44.485676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485532 4711 cpu_manager.go:414] "containerMap: removing container" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" Nov 03 20:18:44.485676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485537 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" Nov 03 20:18:44.485676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485557 4711 cpu_manager.go:414] "containerMap: removing container" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" Nov 03 20:18:44.485676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485562 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" Nov 03 20:18:44.485676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485648 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" Nov 03 20:18:44.485676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485659 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" Nov 03 20:18:44.485676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485665 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="86609111-8d0d-4973-b295-1ac6bb633307" containerName="istio-proxy" Nov 03 20:18:44.485676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.485674 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="300303d0-4e20-4329-a848-79d00b886977" containerName="istio-proxy" Nov 03 20:18:44.490995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.490976 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.508759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.508722 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt"] Nov 03 20:18:44.647641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.647550 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.648050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.647663 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.648050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.647724 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.648050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.647832 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.648050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.647912 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.648050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.647948 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.648050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.647967 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.648270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.648066 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.648270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.648150 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4stdh\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-kube-api-access-4stdh\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.749535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.749502 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.749766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.749573 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.749766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.749625 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.749766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.749658 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.749766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.749682 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.749766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.749718 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.750032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.749814 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4stdh\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-kube-api-access-4stdh\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.750032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.749885 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.750032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.749947 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.750202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.750134 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.750202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.750164 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.750581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.750543 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.750581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.750546 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.751208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.751182 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.752092 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.752068 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.752319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.752301 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.762145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.762118 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4stdh\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-kube-api-access-4stdh\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.762145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.762128 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-929nt\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:44.806072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:44.806043 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:45.120491 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:45.111074 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt"] Nov 03 20:18:45.120491 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:18:45.117987 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85954b98_68ed_4e4f_8ba2_bd7c6dcf28c9.slice/crio-e702b6ec75554b6cd0e07caa703889206220f2f36d42a7301028a55fafa82bd9 WatchSource:0}: Error finding container e702b6ec75554b6cd0e07caa703889206220f2f36d42a7301028a55fafa82bd9: Status 404 returned error can't find the container with id e702b6ec75554b6cd0e07caa703889206220f2f36d42a7301028a55fafa82bd9 Nov 03 20:18:45.121313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:45.121268 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:45.121423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:45.121376 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:45.121490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:45.121464 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:45.259400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:45.259354 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:45.921936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:45.921902 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" event={"ID":"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9","Type":"ContainerStarted","Data":"924a4ed0b66659cffc6980a9f4c501723428da276c67e5287f16fd86bc183cac"} Nov 03 20:18:45.922379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:45.921960 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" event={"ID":"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9","Type":"ContainerStarted","Data":"e702b6ec75554b6cd0e07caa703889206220f2f36d42a7301028a55fafa82bd9"} Nov 03 20:18:45.951651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:45.951594 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" podStartSLOduration=1.9515784250000001 podStartE2EDuration="1.951578425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:45.948794711 +0000 UTC m=+1029.398900758" watchObservedRunningTime="2026-11-03 20:18:45.951578425 +0000 UTC m=+1029.401684472" Nov 03 20:18:46.014792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:46.014255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:46.806421 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:46.806391 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:46.808250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:46.808226 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-929nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:18:46.808382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:46.808269 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:18:46.938085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:46.937984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:47.533398 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:18:47.533361 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:18:47.812536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:47.811033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:47.812536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:47.811588 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-929nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:18:47.812536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:47.811643 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:18:48.644889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:48.644849 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:48.806694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:48.806661 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-929nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:18:48.806861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:48.806726 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:18:49.603615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:49.603585 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:49.810488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:49.807176 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-929nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:18:49.810488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:49.807234 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:18:50.434880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:50.434840 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:50.810874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:50.810836 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-929nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:18:50.811417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:50.811384 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:18:51.443144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:51.443108 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:51.806703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:51.806665 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-929nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:18:51.806920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:51.806721 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:18:52.297585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:52.297543 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:52.843537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:52.843497 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:52.843814 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:52.843785 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:52.845104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:52.845079 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:18:53.101874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:53.101741 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:53.779768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:53.779728 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:54.498384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:54.498333 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:55.193424 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:55.193384 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:55.866383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:55.866351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:55.993431 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:55.993393 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:56.152067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:56.151977 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:56.527675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:56.527634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:57.410385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:57.410347 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:58.255218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:58.255176 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:59.396401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.396360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:18:59.518003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.517959 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9"] Nov 03 20:18:59.518740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.518720 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75b932a-f2a3-43e1-a25a-6efdf2da8740" containerName="istio-proxy" Nov 03 20:18:59.522360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.522337 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.537534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.537506 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9"] Nov 03 20:18:59.594209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.594169 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.594409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.594256 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.594409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.594324 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.594595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.594482 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtmd\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-kube-api-access-zrtmd\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.594595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.594577 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.594706 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.594646 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.594756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.594731 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.594796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.594782 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.594842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.594798 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.695839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.695748 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.695839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.695798 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.695877 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.695932 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.695969 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.696092 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtmd\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-kube-api-access-zrtmd\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.696142 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.696169 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.696216 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.696275 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.696741 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.696923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.696837 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.697036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.697015 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.697036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.697022 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.698674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.698653 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.698773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.698720 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.708432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.708399 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.708570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.708549 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtmd\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-kube-api-access-zrtmd\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:18:59.840568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:18:59.840526 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:19:00.130721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:00.130684 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9"] Nov 03 20:19:00.131332 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:19:00.131304 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b598ce1_a56b_42c6_b34e_4054d7fc290c.slice/crio-9f2e0c0814c736083fea5067cc496e40254943dc66f137a83563c7f67f8942b0 WatchSource:0}: Error finding container 9f2e0c0814c736083fea5067cc496e40254943dc66f137a83563c7f67f8942b0: Status 404 returned error can't find the container with id 9f2e0c0814c736083fea5067cc496e40254943dc66f137a83563c7f67f8942b0 Nov 03 20:19:00.134159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:00.134090 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:00.134258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:00.134203 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:00.134258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:00.134251 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:00.194496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:00.194467 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:00.394828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:00.394732 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:00.831343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:00.831305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:00.966472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:00.965763 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:01.098830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:01.098732 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" event={"ID":"6b598ce1-a56b-42c6-b34e-4054d7fc290c","Type":"ContainerStarted","Data":"18305c90fd035110ad9f958b68ff2a84cd22753138d857681f773462927aad03"} Nov 03 20:19:01.098830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:01.098796 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" event={"ID":"6b598ce1-a56b-42c6-b34e-4054d7fc290c","Type":"ContainerStarted","Data":"9f2e0c0814c736083fea5067cc496e40254943dc66f137a83563c7f67f8942b0"} Nov 03 20:19:01.125699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:01.125646 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" podStartSLOduration=2.12563227 podStartE2EDuration="2.12563227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:01.123764 +0000 UTC m=+1044.573870047" watchObservedRunningTime="2026-11-03 20:19:01.12563227 +0000 UTC m=+1044.575738316" Nov 03 20:19:01.608572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:01.608538 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:01.841328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:01.841298 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:19:01.843062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:01.843032 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:19:01.843239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:01.843081 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:19:01.958996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:01.958907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:02.298699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:02.298668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:02.841822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:02.841791 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:19:02.842206 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:02.841840 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:19:02.992106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:02.992069 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:03.699186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:03.699144 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:03.841171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:03.841138 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:19:03.841380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:03.841194 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:19:04.392144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:04.392108 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:04.841677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:04.841644 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:19:04.841863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:04.841692 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:19:05.090481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:05.090179 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:05.841093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:05.841050 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:19:05.841512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:05.841150 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:19:05.892091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:05.892053 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:06.685830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:06.685795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:06.812353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:06.812313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:06.841230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:06.841195 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 03 20:19:06.841781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:06.841250 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 03 20:19:07.423343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:07.423298 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:07.604230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:07.604195 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc"] Nov 03 20:19:07.693672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:07.693577 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:07.818834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:07.818795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:07.846173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:07.846148 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:19:07.846665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:07.846642 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:19:07.847570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:07.847545 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:19:08.602211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:08.602172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:08.929508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:08.929403 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:09.117096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:09.117064 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:09.242859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:09.242757 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:09.370073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:09.370018 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:09.985675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:09.985630 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:10.645276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:10.645231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:11.323269 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:11.323229 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:11.950364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:11.950327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:12.267011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:12.266942 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:12.605820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:12.605727 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" podUID="ff857a31-27e0-4e2b-81c5-d21795202723" containerName="payload-pre-processing" containerID="cri-o://90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e" gracePeriod=25 Nov 03 20:19:12.901704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:12.901673 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:19:13.059992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.059944 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:13.089525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.087348 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff857a31-27e0-4e2b-81c5-d21795202723-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff857a31-27e0-4e2b-81c5-d21795202723-plugins-config-volume\") pod \"ff857a31-27e0-4e2b-81c5-d21795202723\" (UID: \"ff857a31-27e0-4e2b-81c5-d21795202723\") " Nov 03 20:19:13.094468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.091743 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff857a31-27e0-4e2b-81c5-d21795202723-kube-api-access-9bqs2\" (UniqueName: \"kubernetes.io/projected/ff857a31-27e0-4e2b-81c5-d21795202723-kube-api-access-9bqs2\") pod \"ff857a31-27e0-4e2b-81c5-d21795202723\" (UID: \"ff857a31-27e0-4e2b-81c5-d21795202723\") " Nov 03 20:19:13.094468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.093627 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff857a31-27e0-4e2b-81c5-d21795202723-plugins-config-volume" pod "ff857a31-27e0-4e2b-81c5-d21795202723" (UID: "ff857a31-27e0-4e2b-81c5-d21795202723"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:13.105717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.103945 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff857a31-27e0-4e2b-81c5-d21795202723-kube-api-access-9bqs2" pod "ff857a31-27e0-4e2b-81c5-d21795202723" (UID: "ff857a31-27e0-4e2b-81c5-d21795202723"). InnerVolumeSpecName "kube-api-access-9bqs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:13.194558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.194437 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bqs2\" (UniqueName: \"kubernetes.io/projected/ff857a31-27e0-4e2b-81c5-d21795202723-kube-api-access-9bqs2\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:13.194558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.194501 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff857a31-27e0-4e2b-81c5-d21795202723-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:13.215577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.215542 4711 generic.go:356] "Generic (PLEG): container finished" podID="ff857a31-27e0-4e2b-81c5-d21795202723" containerID="90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e" exitCode=0 Nov 03 20:19:13.215747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.215592 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" Nov 03 20:19:13.215747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.215627 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" event={"ID":"ff857a31-27e0-4e2b-81c5-d21795202723","Type":"ContainerDied","Data":"90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e"} Nov 03 20:19:13.215747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.215692 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc" event={"ID":"ff857a31-27e0-4e2b-81c5-d21795202723","Type":"ContainerDied","Data":"7c2809be58820c4baf1791366f982fb7abc608a266ae303137ef71d586d4f41e"} Nov 03 20:19:13.215747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.215730 4711 scope.go:122] "RemoveContainer" containerID="90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e" Nov 03 20:19:13.226491 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.226462 4711 scope.go:122] "RemoveContainer" containerID="90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e" Nov 03 20:19:13.226765 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:19:13.226746 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e\": container with ID starting with 90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e not found: ID does not exist" containerID="90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e" Nov 03 20:19:13.226815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.226773 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e"} err="failed to get container status \"90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e\": rpc error: code = NotFound desc = could not find container \"90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e\": container with ID starting with 90289318a3fa70355866748b1b6ba221420563c3cc2ccbb649725626a5568c7e not found: ID does not exist" Nov 03 20:19:13.240977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.240943 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc"] Nov 03 20:19:13.248495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.248466 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98148a-5b479ccd4c-dhllc"] Nov 03 20:19:13.849169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:13.849115 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:14.045859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:14.045824 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:14.729076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:14.729039 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff857a31-27e0-4e2b-81c5-d21795202723" path="/var/lib/kubelet/pods/ff857a31-27e0-4e2b-81c5-d21795202723/volumes" Nov 03 20:19:15.213388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:15.213347 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:15.586472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:15.583922 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:16.055370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:16.055317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:16.406530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:16.402775 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:16.603473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:16.596214 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:17.093030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:17.092994 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:17.410525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:17.410374 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:17.534895 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:19:17.534857 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:19:17.541704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:17.541666 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:18.362797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:18.362755 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:18.944978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:18.944930 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:20.108176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:20.108059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:21.107365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:21.107327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:21.386886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:21.386804 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:21.522916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:21.522876 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:21.994587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:21.994550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:22.591438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:22.591402 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:22.768741 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:22.768604 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:22.901624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:22.901533 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:23.667754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:23.667710 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:23.803668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:23.803635 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:24.828602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:24.828565 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:24.985760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:24.985730 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:25.599474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:25.599412 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:26.309231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:26.309158 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:27.173200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:27.173161 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:27.925156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:27.925119 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:28.053083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:28.053042 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:28.227413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:28.227320 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:28.805695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:28.805662 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:29.493921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:29.493889 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:30.305566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:30.305527 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:30.973209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:30.973169 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:31.103901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:31.103866 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:31.267038 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:31.267005 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:31.623850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:31.623513 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:31.972393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:31.972307 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:32.491440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:32.491406 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:33.402951 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:33.402909 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:34.121177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:34.121132 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:34.808904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:34.808836 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:35.612363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:35.612319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:36.302440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:36.302399 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:36.992587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:36.992544 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:37.655762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:37.655712 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:38.318120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:38.318080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:39.090612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:39.090565 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:39.909330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:39.909276 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:40.673800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:40.673761 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:41.352147 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:41.352113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:42.510726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:42.510688 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:42.796768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:42.796679 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8"] Nov 03 20:19:42.906505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:42.906467 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:43.029966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:43.029935 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:43.601211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:43.601163 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:44.394340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:44.394304 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:44.521739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:44.521703 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:44.959959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:44.959928 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:45.596182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:45.596127 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:45.836108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:45.836069 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:46.535359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:46.535131 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:46.665419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:46.665381 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:47.290197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:47.290165 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:47.450216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:47.450153 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:47.536242 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:19:47.536202 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:19:47.751765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:47.751685 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:47.797119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:47.797082 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" podUID="00fdaa29-9500-4fbc-a1f0-077f4ff15733" containerName="payload-processing" containerID="cri-o://dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c" gracePeriod=25 Nov 03 20:19:48.086750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.086716 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:48.287068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.287039 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:19:48.301980 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.301955 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00fdaa29-9500-4fbc-a1f0-077f4ff15733-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fdaa29-9500-4fbc-a1f0-077f4ff15733-plugins-config-volume\") pod \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\" (UID: \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\") " Nov 03 20:19:48.302120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.302058 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fdaa29-9500-4fbc-a1f0-077f4ff15733-kube-api-access-shzxv\" (UniqueName: \"kubernetes.io/projected/00fdaa29-9500-4fbc-a1f0-077f4ff15733-kube-api-access-shzxv\") pod \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\" (UID: \"00fdaa29-9500-4fbc-a1f0-077f4ff15733\") " Nov 03 20:19:48.302369 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.302344 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00fdaa29-9500-4fbc-a1f0-077f4ff15733-plugins-config-volume" pod "00fdaa29-9500-4fbc-a1f0-077f4ff15733" (UID: "00fdaa29-9500-4fbc-a1f0-077f4ff15733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:19:48.302758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.302740 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00fdaa29-9500-4fbc-a1f0-077f4ff15733-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:48.304084 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.304067 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fdaa29-9500-4fbc-a1f0-077f4ff15733-kube-api-access-shzxv" pod "00fdaa29-9500-4fbc-a1f0-077f4ff15733" (UID: "00fdaa29-9500-4fbc-a1f0-077f4ff15733"). InnerVolumeSpecName "kube-api-access-shzxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:19:48.404411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.404336 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shzxv\" (UniqueName: \"kubernetes.io/projected/00fdaa29-9500-4fbc-a1f0-077f4ff15733-kube-api-access-shzxv\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:19:48.698981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.698950 4711 generic.go:356] "Generic (PLEG): container finished" podID="00fdaa29-9500-4fbc-a1f0-077f4ff15733" containerID="dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c" exitCode=0 Nov 03 20:19:48.699321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.699009 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" Nov 03 20:19:48.699321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.699033 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" event={"ID":"00fdaa29-9500-4fbc-a1f0-077f4ff15733","Type":"ContainerDied","Data":"dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c"} Nov 03 20:19:48.699321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.699085 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8" event={"ID":"00fdaa29-9500-4fbc-a1f0-077f4ff15733","Type":"ContainerDied","Data":"cc7ea2dd17726cc9b0a57cb99e01f1678951743329676286498e5d1ef92c6148"} Nov 03 20:19:48.699321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.699114 4711 scope.go:122] "RemoveContainer" containerID="dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c" Nov 03 20:19:48.709777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.709759 4711 scope.go:122] "RemoveContainer" containerID="dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c" Nov 03 20:19:48.710043 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:19:48.710021 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c\": container with ID starting with dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c not found: ID does not exist" containerID="dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c" Nov 03 20:19:48.710114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.710055 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c"} err="failed to get container status \"dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c\": rpc error: code = NotFound desc = could not find container \"dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c\": container with ID starting with dcea74e4568b7370adf19f5fd4b70a3a6959c6560867167b25eac75252b91d2c not found: ID does not exist" Nov 03 20:19:48.721021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.720988 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8"] Nov 03 20:19:48.724495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.724462 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9b65143e-765c4447d-lk6s8"] Nov 03 20:19:48.765838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:48.765796 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:49.095358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:49.095308 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:49.231204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:49.231173 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:49.714338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:49.714298 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:50.312580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:50.312540 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:50.456348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:50.456313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:50.682271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:50.682193 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fdaa29-9500-4fbc-a1f0-077f4ff15733" path="/var/lib/kubelet/pods/00fdaa29-9500-4fbc-a1f0-077f4ff15733/volumes" Nov 03 20:19:50.802884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:50.802851 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:51.631716 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:51.631680 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:51.853889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:51.853852 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:51.988965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:51.988881 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:52.395431 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:52.395397 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:52.688540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:52.688440 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:53.110076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:53.110037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:53.259098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:53.259063 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:53.406526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:53.402334 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:54.252022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:54.251985 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:55.294347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:55.294311 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:55.917852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:55.917818 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:56.043528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.043495 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:56.191684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.191602 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:56.223551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.223520 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48"] Nov 03 20:19:56.224614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.224587 4711 cpu_manager.go:414] "containerMap: removing container" podUID="00fdaa29-9500-4fbc-a1f0-077f4ff15733" containerName="payload-processing" Nov 03 20:19:56.224614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.224611 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fdaa29-9500-4fbc-a1f0-077f4ff15733" containerName="payload-processing" Nov 03 20:19:56.224801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.224660 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ff857a31-27e0-4e2b-81c5-d21795202723" containerName="payload-pre-processing" Nov 03 20:19:56.224801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.224669 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff857a31-27e0-4e2b-81c5-d21795202723" containerName="payload-pre-processing" Nov 03 20:19:56.224876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.224848 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fdaa29-9500-4fbc-a1f0-077f4ff15733" containerName="payload-processing" Nov 03 20:19:56.224876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.224869 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff857a31-27e0-4e2b-81c5-d21795202723" containerName="payload-pre-processing" Nov 03 20:19:56.229350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.229332 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.237903 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.237863 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6b9849-serving-cert\"" Nov 03 20:19:56.242201 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.242116 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48"] Nov 03 20:19:56.331203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.331167 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:56.380657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.379472 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.380657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.379536 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-metrics-certs\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.380657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.379781 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgnt\" (UniqueName: \"kubernetes.io/projected/02d20df3-65b8-4cd0-bf6c-62d540e85716-kube-api-access-wbgnt\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.380657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.379893 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d20df3-65b8-4cd0-bf6c-62d540e85716-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.458824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.458718 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:56.482177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.482147 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d20df3-65b8-4cd0-bf6c-62d540e85716-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.482345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.482301 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.482345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.482334 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-metrics-certs\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.482437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.482429 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgnt\" (UniqueName: \"kubernetes.io/projected/02d20df3-65b8-4cd0-bf6c-62d540e85716-kube-api-access-wbgnt\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.482511 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:19:56.482479 4711 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-serving-cert: secret "maas-api-e2e-ipp-a-6b9849-serving-cert" not found Nov 03 20:19:56.482567 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:19:56.482548 4711 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls podName:02d20df3-65b8-4cd0-bf6c-62d540e85716 nodeName:}" failed. No retries permitted until 2026-11-03 20:19:56.982526509 +0000 UTC m=+1100.432632548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls") pod "maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" (UID: "02d20df3-65b8-4cd0-bf6c-62d540e85716") : secret "maas-api-e2e-ipp-a-6b9849-serving-cert" not found Nov 03 20:19:56.482821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.482800 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d20df3-65b8-4cd0-bf6c-62d540e85716-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.484932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.484911 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-metrics-certs\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.497460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.497430 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgnt\" (UniqueName: \"kubernetes.io/projected/02d20df3-65b8-4cd0-bf6c-62d540e85716-kube-api-access-wbgnt\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:56.911309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.911263 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:56.990623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:56.990573 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:57.010364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:57.010332 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:57.169780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:57.160778 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:57.202584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:57.202554 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:57.344651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:57.344600 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:57.481357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:57.481276 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48"] Nov 03 20:19:57.769758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:57.769684 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" event={"ID":"02d20df3-65b8-4cd0-bf6c-62d540e85716","Type":"ContainerStarted","Data":"5600adabb2be2dd26da4630c762deb2fcc6ab4e8b1054ead63aef6c1444e47cc"} Nov 03 20:19:57.855061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:57.855019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:19:58.777086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:58.777046 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" event={"ID":"02d20df3-65b8-4cd0-bf6c-62d540e85716","Type":"ContainerStarted","Data":"cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21"} Nov 03 20:19:58.777551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:58.777114 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:19:58.802196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:58.802151 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" podStartSLOduration=2.390500902 podStartE2EDuration="2.802138354s" totalImagesPullingTime="411.637452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:58.79913886 +0000 UTC m=+1102.249244908" watchObservedRunningTime="2026-11-03 20:19:58.802138354 +0000 UTC m=+1102.252244426" Nov 03 20:19:59.179847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:19:59.179754 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:00.191180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:00.191141 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:00.332268 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:00.332231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:01.534212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:01.534175 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:02.069062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.069029 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7"] Nov 03 20:20:02.069361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.069331 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" podUID="d14f1170-a9d8-4495-999c-1f60407db883" containerName="maas-api" containerID="cri-o://f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72" gracePeriod=30 Nov 03 20:20:02.153644 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.153612 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:02.264077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.262615 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" podUID="d14f1170-a9d8-4495-999c-1f60407db883" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.91:8443/health\": dial tcp 10.133.0.91:8443: connect: connection refused" Nov 03 20:20:02.443542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.437320 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:02.506261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.506231 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:20:02.569227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.569188 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:02.638961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.638928 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-maas-api-tls\") pod \"d14f1170-a9d8-4495-999c-1f60407db883\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " Nov 03 20:20:02.639161 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.638997 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14f1170-a9d8-4495-999c-1f60407db883-kube-api-access-5cwrz\" (UniqueName: \"kubernetes.io/projected/d14f1170-a9d8-4495-999c-1f60407db883-kube-api-access-5cwrz\") pod \"d14f1170-a9d8-4495-999c-1f60407db883\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " Nov 03 20:20:02.639161 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.639042 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-metrics-certs\") pod \"d14f1170-a9d8-4495-999c-1f60407db883\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " Nov 03 20:20:02.639161 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.639117 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14f1170-a9d8-4495-999c-1f60407db883-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f1170-a9d8-4495-999c-1f60407db883-openshift-service-ca\") pod \"d14f1170-a9d8-4495-999c-1f60407db883\" (UID: \"d14f1170-a9d8-4495-999c-1f60407db883\") " Nov 03 20:20:02.639568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.639536 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14f1170-a9d8-4495-999c-1f60407db883-openshift-service-ca" pod "d14f1170-a9d8-4495-999c-1f60407db883" (UID: "d14f1170-a9d8-4495-999c-1f60407db883"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:02.639947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.639928 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d14f1170-a9d8-4495-999c-1f60407db883-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:02.641235 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.641205 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-maas-api-tls" pod "d14f1170-a9d8-4495-999c-1f60407db883" (UID: "d14f1170-a9d8-4495-999c-1f60407db883"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:02.641235 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.641220 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-metrics-certs" pod "d14f1170-a9d8-4495-999c-1f60407db883" (UID: "d14f1170-a9d8-4495-999c-1f60407db883"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:02.641375 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.641268 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14f1170-a9d8-4495-999c-1f60407db883-kube-api-access-5cwrz" pod "d14f1170-a9d8-4495-999c-1f60407db883" (UID: "d14f1170-a9d8-4495-999c-1f60407db883"). InnerVolumeSpecName "kube-api-access-5cwrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:02.700618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.700400 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:02.741928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.741876 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:02.741928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.741910 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cwrz\" (UniqueName: \"kubernetes.io/projected/d14f1170-a9d8-4495-999c-1f60407db883-kube-api-access-5cwrz\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:02.741928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.741919 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14f1170-a9d8-4495-999c-1f60407db883-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:02.824153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.824121 4711 generic.go:356] "Generic (PLEG): container finished" podID="d14f1170-a9d8-4495-999c-1f60407db883" containerID="f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72" exitCode=0 Nov 03 20:20:02.824328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.824179 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" Nov 03 20:20:02.824328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.824214 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" event={"ID":"d14f1170-a9d8-4495-999c-1f60407db883","Type":"ContainerDied","Data":"f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72"} Nov 03 20:20:02.824328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.824290 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7" event={"ID":"d14f1170-a9d8-4495-999c-1f60407db883","Type":"ContainerDied","Data":"df3edbb0e5c24c3cdc3cfa3d45f93138042d4945b7876b3a666fcc36d006fc81"} Nov 03 20:20:02.824328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.824314 4711 scope.go:122] "RemoveContainer" containerID="f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72" Nov 03 20:20:02.835768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.835749 4711 scope.go:122] "RemoveContainer" containerID="f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72" Nov 03 20:20:02.836049 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:20:02.836028 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72\": container with ID starting with f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72 not found: ID does not exist" containerID="f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72" Nov 03 20:20:02.836115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.836064 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72"} err="failed to get container status \"f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72\": rpc error: code = NotFound desc = could not find container \"f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72\": container with ID starting with f98febde858f02ddc17c13aa7a4734844a659f282cbb9f075f499ecaa232ce72 not found: ID does not exist" Nov 03 20:20:02.845498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.845436 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7"] Nov 03 20:20:02.848655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:02.848636 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9b65143e-54c4bcff7-nbbd7"] Nov 03 20:20:03.594897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:03.594835 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:04.253141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.253099 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:04.681648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.681617 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14f1170-a9d8-4495-999c-1f60407db883" path="/var/lib/kubelet/pods/d14f1170-a9d8-4495-999c-1f60407db883/volumes" Nov 03 20:20:04.790322 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.790293 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:20:04.831021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.830984 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5"] Nov 03 20:20:04.832086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.832060 4711 cpu_manager.go:414] "containerMap: removing container" podUID="d14f1170-a9d8-4495-999c-1f60407db883" containerName="maas-api" Nov 03 20:20:04.832086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.832085 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14f1170-a9d8-4495-999c-1f60407db883" containerName="maas-api" Nov 03 20:20:04.832287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.832273 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14f1170-a9d8-4495-999c-1f60407db883" containerName="maas-api" Nov 03 20:20:04.843494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.843469 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:04.859541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.859497 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5"] Nov 03 20:20:04.876021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.874614 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1b7b5e-serving-cert\"" Nov 03 20:20:04.916882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.916849 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:04.987559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.987477 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:04.987559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.987531 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-metrics-certs\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:04.987806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.987779 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzrcb\" (UniqueName: \"kubernetes.io/projected/6fa0a87a-b607-4444-8022-6c5db411a74e-kube-api-access-hzrcb\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:04.987978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:04.987959 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa0a87a-b607-4444-8022-6c5db411a74e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.056398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.056351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:05.089516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.089485 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.089715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.089524 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-metrics-certs\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.089715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.089583 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzrcb\" (UniqueName: \"kubernetes.io/projected/6fa0a87a-b607-4444-8022-6c5db411a74e-kube-api-access-hzrcb\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.089827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.089764 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa0a87a-b607-4444-8022-6c5db411a74e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.090393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.090366 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa0a87a-b607-4444-8022-6c5db411a74e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.091978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.091949 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.092081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.092025 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-metrics-certs\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.102164 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.102143 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzrcb\" (UniqueName: \"kubernetes.io/projected/6fa0a87a-b607-4444-8022-6c5db411a74e-kube-api-access-hzrcb\") pod \"maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.182804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.182774 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:05.503809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.503778 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5"] Nov 03 20:20:05.835206 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.835169 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:05.867541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:05.867510 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" event={"ID":"6fa0a87a-b607-4444-8022-6c5db411a74e","Type":"ContainerStarted","Data":"a1f6003ce76ec93572b136d5d13e0644823f809d6aba2462fe4a4d7f0fbeaab6"} Nov 03 20:20:06.639520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:06.639479 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:06.775138 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:06.775102 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:06.876863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:06.876822 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" event={"ID":"6fa0a87a-b607-4444-8022-6c5db411a74e","Type":"ContainerStarted","Data":"b2446e1b5bfa9d156f7c065ae6fdc7d3da744c70f654bc063c4fc493a7902cb5"} Nov 03 20:20:06.877324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:06.876914 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:06.901052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:06.900960 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" podStartSLOduration=2.449398977 podStartE2EDuration="2.900947156s" totalImagesPullingTime="451.548179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:06.89713442 +0000 UTC m=+1110.347240468" watchObservedRunningTime="2026-11-03 20:20:06.900947156 +0000 UTC m=+1110.351053202" Nov 03 20:20:07.956425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:07.956367 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:08.799583 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:08.799542 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:09.104041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:09.103952 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:09.223050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:09.223003 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:09.367528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:09.367420 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:09.695756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:09.695667 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:09.824251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:09.824161 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:10.032532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:10.032500 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:10.161260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:10.161224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:10.861340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:10.861281 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:11.808117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:11.808080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:11.937284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:11.937245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:12.074437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:12.074341 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:12.713484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:12.713430 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:12.892206 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:12.892168 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:20:13.478886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:13.478848 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:13.615083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:13.615030 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:14.199767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:14.199723 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:15.018296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:15.018228 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:15.983260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:15.983219 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:16.943071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:16.943035 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:17.583185 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:20:17.583146 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:20:17.711575 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:17.711537 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:17.841365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:17.841270 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:18.619971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:18.619921 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:18.863727 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:18.863690 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:18.992096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:18.991962 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:20.104030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:20.103985 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:20.234958 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:20.234909 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:20.921815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:20.921765 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:21.141709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:21.141673 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:21.280877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:21.280840 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:21.480738 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:21.480692 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:22.145543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:22.145505 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:23.113900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:23.113866 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:24.436870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:24.436831 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:25.496642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:25.496601 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:26.301842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:26.301808 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:26.439869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:26.439814 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:27.121779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:27.121744 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:27.621324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:27.621269 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:27.755497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:27.755466 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:28.306516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:28.306477 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:29.061517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:29.061471 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:29.916440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:29.916401 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:30.081092 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:30.081042 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:30.471902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:30.471858 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:31.193898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:31.193855 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:31.395012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:31.394975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:31.644429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:31.644388 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:32.208361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:32.208325 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:32.340734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:32.340691 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:32.695363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:32.695329 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:32.822339 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:32.822301 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:33.351081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:33.351047 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:33.613011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:33.612917 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:33.948866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:33.948786 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:34.752719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:34.752676 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:35.082666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:35.082635 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:35.212073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:35.212036 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:35.485537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:35.485411 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:35.619035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:35.618995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:36.574578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.574476 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:36.749888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.749805 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:36.753119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.753089 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5"] Nov 03 20:20:36.757638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.757618 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:36.762348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.761910 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f2218a\"" Nov 03 20:20:36.763273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.763248 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f2218a-dockercfg-wsc98\"" Nov 03 20:20:36.768672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.768649 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5"] Nov 03 20:20:36.811860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.811830 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjxml\" (UniqueName: \"kubernetes.io/projected/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-kube-api-access-xjxml\") pod \"payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5\" (UID: \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:36.812086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.811868 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5\" (UID: \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:36.891786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.891685 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:36.916968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.916926 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5\" (UID: \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:36.917520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.917493 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjxml\" (UniqueName: \"kubernetes.io/projected/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-kube-api-access-xjxml\") pod \"payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5\" (UID: \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:36.919397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.918484 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5\" (UID: \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:36.967190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:36.967150 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjxml\" (UniqueName: \"kubernetes.io/projected/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-kube-api-access-xjxml\") pod \"payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5\" (UID: \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:37.077335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:37.077303 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:37.366989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:37.366958 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5"] Nov 03 20:20:37.367569 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:20:37.367545 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3b3d29_df89_41e0_8fbc_fe45ad54a366.slice/crio-ea1f1d419cead4436323b93ece47b65302f9c5f7932968932daf28aa70dff1ae WatchSource:0}: Error finding container ea1f1d419cead4436323b93ece47b65302f9c5f7932968932daf28aa70dff1ae: Status 404 returned error can't find the container with id ea1f1d419cead4436323b93ece47b65302f9c5f7932968932daf28aa70dff1ae Nov 03 20:20:37.763856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:37.763818 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:38.290296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:38.290252 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" event={"ID":"3e3b3d29-df89-41e0-8fbc-fe45ad54a366","Type":"ContainerStarted","Data":"1cc8d20a565f9e534e909291056e64bdb26da0de522d638b24b6bf8fa69a42c5"} Nov 03 20:20:38.290536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:38.290352 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:38.290536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:38.290374 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" event={"ID":"3e3b3d29-df89-41e0-8fbc-fe45ad54a366","Type":"ContainerStarted","Data":"ea1f1d419cead4436323b93ece47b65302f9c5f7932968932daf28aa70dff1ae"} Nov 03 20:20:38.312000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:38.311950 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" podStartSLOduration=2.311935796 podStartE2EDuration="2.311935796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:38.308743065 +0000 UTC m=+1141.758849113" watchObservedRunningTime="2026-11-03 20:20:38.311935796 +0000 UTC m=+1141.762041843" Nov 03 20:20:38.858106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:38.858061 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:39.789046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:39.789002 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:40.517946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:40.517906 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:40.646282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:40.646222 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:41.171234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:41.171194 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:41.335408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:41.335374 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:41.605030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:41.604982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:41.763465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:41.763408 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:42.364686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:42.364652 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:42.630185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:42.630088 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:43.741134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:43.741094 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:44.518543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:44.518503 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:44.639623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:44.639558 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:44.791648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:44.791552 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:46.859903 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:46.859863 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:47.064112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:47.064072 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:47.205077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:47.204946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:47.545920 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:20:47.545885 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:20:47.905991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:47.905892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:48.037359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:48.037312 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:48.483553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:48.483485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:48.614942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:48.614906 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:48.922432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:48.922396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:49.046049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:49.046013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:49.242497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:49.242362 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:49.304887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:49.304861 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:20:49.487346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:49.487316 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:49.825866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:49.825827 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:50.744355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:50.744321 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:51.030746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:51.030706 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:51.327649 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:51.327562 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:51.483930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:51.483892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:51.638864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:51.638771 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:51.772463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:51.772423 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:52.246840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.246795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:52.596636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.596603 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:52.669312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.669281 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm"] Nov 03 20:20:52.673846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.673814 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:20:52.680316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.680295 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d6789f-dockercfg-29lqn\"" Nov 03 20:20:52.687317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.687293 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm"] Nov 03 20:20:52.711609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.711586 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d6789f\"" Nov 03 20:20:52.814971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.814933 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncp7l\" (UniqueName: \"kubernetes.io/projected/54d2a0d6-60ee-460b-84f2-29f33842cf08-kube-api-access-ncp7l\") pod \"payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm\" (UID: \"54d2a0d6-60ee-460b-84f2-29f33842cf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:20:52.815597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.815560 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d2a0d6-60ee-460b-84f2-29f33842cf08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm\" (UID: \"54d2a0d6-60ee-460b-84f2-29f33842cf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:20:52.821218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.821189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:52.918162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.918068 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncp7l\" (UniqueName: \"kubernetes.io/projected/54d2a0d6-60ee-460b-84f2-29f33842cf08-kube-api-access-ncp7l\") pod \"payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm\" (UID: \"54d2a0d6-60ee-460b-84f2-29f33842cf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:20:52.918162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.918134 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d2a0d6-60ee-460b-84f2-29f33842cf08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm\" (UID: \"54d2a0d6-60ee-460b-84f2-29f33842cf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:20:52.920173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.919177 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d2a0d6-60ee-460b-84f2-29f33842cf08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm\" (UID: \"54d2a0d6-60ee-460b-84f2-29f33842cf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:20:52.931203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.931176 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncp7l\" (UniqueName: \"kubernetes.io/projected/54d2a0d6-60ee-460b-84f2-29f33842cf08-kube-api-access-ncp7l\") pod \"payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm\" (UID: \"54d2a0d6-60ee-460b-84f2-29f33842cf08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:20:52.993389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:52.993356 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:20:53.221560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:53.221434 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:53.284798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:53.284764 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm"] Nov 03 20:20:53.496388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:53.496299 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" event={"ID":"54d2a0d6-60ee-460b-84f2-29f33842cf08","Type":"ContainerStarted","Data":"0ff31802bcbcc603a7cfce01d1eeec8b299daec91f0b50e995e4820c90462292"} Nov 03 20:20:53.496388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:53.496370 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" event={"ID":"54d2a0d6-60ee-460b-84f2-29f33842cf08","Type":"ContainerStarted","Data":"461d5af4fa3d0bf283590f3c220d78732c2ffc391a367911d3e104dae1d24c24"} Nov 03 20:20:53.504713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:53.496981 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:20:53.523724 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:53.522839 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" podStartSLOduration=1.522815754 podStartE2EDuration="1.522815754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:53.520098236 +0000 UTC m=+1156.970204309" watchObservedRunningTime="2026-11-03 20:20:53.522815754 +0000 UTC m=+1156.972921802" Nov 03 20:20:54.020083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:54.020048 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:54.165549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:54.165509 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:55.059581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:55.059537 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:55.463361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:55.463256 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:55.590687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:55.590649 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:55.996019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:55.995981 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:56.128407 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:56.127534 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:56.328747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:56.328679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:56.699625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:56.699514 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:56.847805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:56.847747 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:57.169946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:57.169908 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:58.155081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:58.155033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:59.100944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:59.100883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:20:59.883146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:20:59.883109 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:00.561851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:00.561814 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:00.685377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:00.685305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:01.261714 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.261677 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:01.598375 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.598293 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn"] Nov 03 20:21:01.604335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.604309 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.615165 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.615139 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d6789f-serving-cert\"" Nov 03 20:21:01.620889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.620858 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn"] Nov 03 20:21:01.708226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.708189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:01.769591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.769541 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/761300fa-3d54-4401-b409-11657e9ea609-kube-api-access-l6k8p\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.769781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.769605 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.769781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.769725 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761300fa-3d54-4401-b409-11657e9ea609-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.770026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.770010 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-metrics-certs\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.840430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.840391 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:01.873826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.873724 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-metrics-certs\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.875398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.875369 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/761300fa-3d54-4401-b409-11657e9ea609-kube-api-access-l6k8p\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.875658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.875639 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.892477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.892068 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761300fa-3d54-4401-b409-11657e9ea609-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.899527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.892766 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-metrics-certs\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.900366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.893314 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761300fa-3d54-4401-b409-11657e9ea609-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.918470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.914381 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.950467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.928150 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/761300fa-3d54-4401-b409-11657e9ea609-kube-api-access-l6k8p\") pod \"maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:01.950467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:01.928428 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:02.304542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:02.304510 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn"] Nov 03 20:21:02.569321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:02.569223 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" event={"ID":"761300fa-3d54-4401-b409-11657e9ea609","Type":"ContainerStarted","Data":"e031328be8df937e9ccdf921bf8d4acf56d403bf4c4e1245574cf619fb0748c9"} Nov 03 20:21:02.802312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:02.802278 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:03.578252 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:03.578218 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" event={"ID":"761300fa-3d54-4401-b409-11657e9ea609","Type":"ContainerStarted","Data":"5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24"} Nov 03 20:21:03.578434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:03.578282 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:03.604472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:03.604405 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" podStartSLOduration=2.220595987 podStartE2EDuration="2.604392233s" totalImagesPullingTime="383.796246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:03.601211283 +0000 UTC m=+1167.051317331" watchObservedRunningTime="2026-11-03 20:21:03.604392233 +0000 UTC m=+1167.054498280" Nov 03 20:21:03.636191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:03.636155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:04.603975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:04.603912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:05.507713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:05.507672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:05.512685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:05.512658 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:21:05.811502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:05.811362 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:05.944318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:05.944275 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:06.194362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.194288 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs"] Nov 03 20:21:06.199259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.199240 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.210911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.210890 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs"] Nov 03 20:21:06.266347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.266319 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.266514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.266373 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-kube-api-access-cltwb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.266571 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.266534 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11439f87-11d4-4015-853c-b5d587d54f6d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.266711 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.266695 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.266759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.266741 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.266834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.266816 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.266872 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.266854 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.368007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.367970 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.368204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.368017 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.368204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.368123 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.368204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.368163 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-kube-api-access-cltwb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.368373 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.368282 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11439f87-11d4-4015-853c-b5d587d54f6d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.368495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.368435 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.368571 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.368520 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.368786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.368757 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.370682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.370662 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.370813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.370782 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.370921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.370862 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11439f87-11d4-4015-853c-b5d587d54f6d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.371051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.371034 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.382612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.382585 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.382790 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.382771 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-kube-api-access-cltwb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x8ffs\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.509477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.509426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:06.516260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.516241 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:06.653129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.649600 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:06.777927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.777835 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:06.856113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.856082 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs"] Nov 03 20:21:06.856506 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:21:06.856359 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11439f87_11d4_4015_853c_b5d587d54f6d.slice/crio-6a530555d62ada599a8d0218f07544fda08f348d9001c43dc4bf293dafe659df WatchSource:0}: Error finding container 6a530555d62ada599a8d0218f07544fda08f348d9001c43dc4bf293dafe659df: Status 404 returned error can't find the container with id 6a530555d62ada599a8d0218f07544fda08f348d9001c43dc4bf293dafe659df Nov 03 20:21:06.859124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.859091 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:06.859210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:06.859152 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:07.078178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:07.078100 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:07.217839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:07.217802 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:07.614540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:07.613373 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" event={"ID":"11439f87-11d4-4015-853c-b5d587d54f6d","Type":"ContainerStarted","Data":"8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64"} Nov 03 20:21:07.614540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:07.613440 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" event={"ID":"11439f87-11d4-4015-853c-b5d587d54f6d","Type":"ContainerStarted","Data":"6a530555d62ada599a8d0218f07544fda08f348d9001c43dc4bf293dafe659df"} Nov 03 20:21:07.614540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:07.613484 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:07.615702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:07.615341 4711 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-x8ffs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:21:07.615702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:07.615396 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" podUID="11439f87-11d4-4015-853c-b5d587d54f6d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:21:07.644971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:07.644908 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" podStartSLOduration=1.644894493 podStartE2EDuration="1.644894493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:07.642919155 +0000 UTC m=+1171.093025203" watchObservedRunningTime="2026-11-03 20:21:07.644894493 +0000 UTC m=+1171.095000539" Nov 03 20:21:07.694477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:07.694263 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:08.604842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:08.604801 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:08.637394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:08.637364 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:21:09.592022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:09.591988 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:21:09.686314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:09.686283 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:10.606567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:10.606529 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:10.972918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:10.972828 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:11.107182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:11.107145 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:11.305527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:11.304816 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:11.705797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:11.705703 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:12.014968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:12.014932 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:12.149038 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:12.148981 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:12.675787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:12.675749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:12.983794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:12.983709 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:13.288204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:13.288168 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:13.953296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:13.953255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:15.212399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:15.212369 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:16.527144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:16.527064 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:17.535480 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:21:17.535403 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:21:17.799597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:17.799503 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:19.148466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:19.145610 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:19.480830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:19.480737 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:20.044302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:20.044265 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:20.184020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:20.183982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:21.137230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:21.137195 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:21.422324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:21.422229 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:21.528197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:21.528155 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm"] Nov 03 20:21:21.638110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:21.638075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:21.767938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:21.767900 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:21.896073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:21.896017 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:22.713781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:22.713734 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:23.404247 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:23.404214 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:24.116386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:24.116349 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:24.745405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:24.745354 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:24.867379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:24.867345 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:25.519885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:25.519844 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:25.654628 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:25.654578 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.031023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.030980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.173482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.173432 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.322531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.314464 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.495806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.495770 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g"] Nov 03 20:21:26.496058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.496032 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" containerID="cri-o://9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22" gracePeriod=30 Nov 03 20:21:26.528540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.528489 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" podUID="54d2a0d6-60ee-460b-84f2-29f33842cf08" containerName="payload-processing" containerID="cri-o://0ff31802bcbcc603a7cfce01d1eeec8b299daec91f0b50e995e4820c90462292" gracePeriod=25 Nov 03 20:21:26.555559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.555530 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.835593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.835562 4711 generic.go:356] "Generic (PLEG): container finished" podID="54d2a0d6-60ee-460b-84f2-29f33842cf08" containerID="0ff31802bcbcc603a7cfce01d1eeec8b299daec91f0b50e995e4820c90462292" exitCode=0 Nov 03 20:21:26.835932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.835911 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" event={"ID":"54d2a0d6-60ee-460b-84f2-29f33842cf08","Type":"ContainerDied","Data":"0ff31802bcbcc603a7cfce01d1eeec8b299daec91f0b50e995e4820c90462292"} Nov 03 20:21:26.885281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.885227 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:26.969336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:26.969311 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:21:27.089955 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.089913 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54d2a0d6-60ee-460b-84f2-29f33842cf08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d2a0d6-60ee-460b-84f2-29f33842cf08-plugins-config-volume\") pod \"54d2a0d6-60ee-460b-84f2-29f33842cf08\" (UID: \"54d2a0d6-60ee-460b-84f2-29f33842cf08\") " Nov 03 20:21:27.090552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.090065 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d2a0d6-60ee-460b-84f2-29f33842cf08-kube-api-access-ncp7l\" (UniqueName: \"kubernetes.io/projected/54d2a0d6-60ee-460b-84f2-29f33842cf08-kube-api-access-ncp7l\") pod \"54d2a0d6-60ee-460b-84f2-29f33842cf08\" (UID: \"54d2a0d6-60ee-460b-84f2-29f33842cf08\") " Nov 03 20:21:27.090552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.090398 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d2a0d6-60ee-460b-84f2-29f33842cf08-plugins-config-volume" pod "54d2a0d6-60ee-460b-84f2-29f33842cf08" (UID: "54d2a0d6-60ee-460b-84f2-29f33842cf08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:27.090745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.090718 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54d2a0d6-60ee-460b-84f2-29f33842cf08-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:27.092685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.092657 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d2a0d6-60ee-460b-84f2-29f33842cf08-kube-api-access-ncp7l" pod "54d2a0d6-60ee-460b-84f2-29f33842cf08" (UID: "54d2a0d6-60ee-460b-84f2-29f33842cf08"). InnerVolumeSpecName "kube-api-access-ncp7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:27.193187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.193089 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncp7l\" (UniqueName: \"kubernetes.io/projected/54d2a0d6-60ee-460b-84f2-29f33842cf08-kube-api-access-ncp7l\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:27.576991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.576930 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:27.810027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.809988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:27.861434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.861340 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" event={"ID":"54d2a0d6-60ee-460b-84f2-29f33842cf08","Type":"ContainerDied","Data":"461d5af4fa3d0bf283590f3c220d78732c2ffc391a367911d3e104dae1d24c24"} Nov 03 20:21:27.861434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.861418 4711 scope.go:122] "RemoveContainer" containerID="0ff31802bcbcc603a7cfce01d1eeec8b299daec91f0b50e995e4820c90462292" Nov 03 20:21:27.861716 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.861614 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm" Nov 03 20:21:27.894073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.894042 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm"] Nov 03 20:21:27.897687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:27.897664 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d6789f-84db6df874-vqsnm"] Nov 03 20:21:28.067230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:28.067181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:28.197664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:28.197568 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:28.683098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:28.683064 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d2a0d6-60ee-460b-84f2-29f33842cf08" path="/var/lib/kubelet/pods/54d2a0d6-60ee-460b-84f2-29f33842cf08/volumes" Nov 03 20:21:28.916208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:28.916170 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:29.801855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:29.801777 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:31.087981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.087945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:31.803565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.803533 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:21:31.821121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.821088 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:31.881514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.881479 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-token\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-token\") pod \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " Nov 03 20:21:31.881700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.881588 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-envoy\") pod \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " Nov 03 20:21:31.881700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.881668 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-credential-socket\") pod \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " Nov 03 20:21:31.881814 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.881773 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istiod-ca-cert\") pod \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " Nov 03 20:21:31.881814 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.881803 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-kube-api-access-bmzph\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-kube-api-access-bmzph\") pod \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " Nov 03 20:21:31.881930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.881899 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-data\") pod \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " Nov 03 20:21:31.882018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.882002 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-podinfo\") pod \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " Nov 03 20:21:31.882096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.882081 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-socket\") pod \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " Nov 03 20:21:31.882159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.882132 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-certs\") pod \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\" (UID: \"fdcdbf31-f878-4515-a5a2-e1b43b1c7420\") " Nov 03 20:21:31.882770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.882490 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-socket" pod "fdcdbf31-f878-4515-a5a2-e1b43b1c7420" (UID: "fdcdbf31-f878-4515-a5a2-e1b43b1c7420"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:31.882770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.882672 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-data" pod "fdcdbf31-f878-4515-a5a2-e1b43b1c7420" (UID: "fdcdbf31-f878-4515-a5a2-e1b43b1c7420"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:31.882770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.882733 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istiod-ca-cert" pod "fdcdbf31-f878-4515-a5a2-e1b43b1c7420" (UID: "fdcdbf31-f878-4515-a5a2-e1b43b1c7420"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:31.883296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.882892 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-certs" pod "fdcdbf31-f878-4515-a5a2-e1b43b1c7420" (UID: "fdcdbf31-f878-4515-a5a2-e1b43b1c7420"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:31.883296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.883259 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.883296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.883286 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.883715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.883300 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.883715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.883313 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.883715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.883508 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-credential-socket" pod "fdcdbf31-f878-4515-a5a2-e1b43b1c7420" (UID: "fdcdbf31-f878-4515-a5a2-e1b43b1c7420"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:31.884231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.884203 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-envoy" pod "fdcdbf31-f878-4515-a5a2-e1b43b1c7420" (UID: "fdcdbf31-f878-4515-a5a2-e1b43b1c7420"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:31.885338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.885318 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-token" pod "fdcdbf31-f878-4515-a5a2-e1b43b1c7420" (UID: "fdcdbf31-f878-4515-a5a2-e1b43b1c7420"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:31.885509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.885433 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-podinfo" pod "fdcdbf31-f878-4515-a5a2-e1b43b1c7420" (UID: "fdcdbf31-f878-4515-a5a2-e1b43b1c7420"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:31.885592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.885570 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-kube-api-access-bmzph" pod "fdcdbf31-f878-4515-a5a2-e1b43b1c7420" (UID: "fdcdbf31-f878-4515-a5a2-e1b43b1c7420"). InnerVolumeSpecName "kube-api-access-bmzph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:31.898461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.898367 4711 generic.go:356] "Generic (PLEG): container finished" podID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerID="9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22" exitCode=0 Nov 03 20:21:31.898461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.898426 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" Nov 03 20:21:31.898597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.898472 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" event={"ID":"fdcdbf31-f878-4515-a5a2-e1b43b1c7420","Type":"ContainerDied","Data":"9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22"} Nov 03 20:21:31.898597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.898565 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g" event={"ID":"fdcdbf31-f878-4515-a5a2-e1b43b1c7420","Type":"ContainerDied","Data":"25e1cb5a109b3b0755ad026e6160901359a08063193e6da3ceab62be04ec418e"} Nov 03 20:21:31.898674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.898614 4711 scope.go:122] "RemoveContainer" containerID="9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22" Nov 03 20:21:31.919701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.919676 4711 scope.go:122] "RemoveContainer" containerID="b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d" Nov 03 20:21:31.930277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.930258 4711 scope.go:122] "RemoveContainer" containerID="9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22" Nov 03 20:21:31.930593 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:21:31.930573 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22\": container with ID starting with 9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22 not found: ID does not exist" containerID="9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22" Nov 03 20:21:31.930677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.930605 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22"} err="failed to get container status \"9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22\": rpc error: code = NotFound desc = could not find container \"9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22\": container with ID starting with 9c5cbb460fde47a2b2b3da54d30dd86b58770ec9fbf41673e5532fac7dd7ff22 not found: ID does not exist" Nov 03 20:21:31.930677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.930629 4711 scope.go:122] "RemoveContainer" containerID="b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d" Nov 03 20:21:31.930899 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:21:31.930882 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d\": container with ID starting with b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d not found: ID does not exist" containerID="b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d" Nov 03 20:21:31.930938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.930906 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d"} err="failed to get container status \"b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d\": rpc error: code = NotFound desc = could not find container \"b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d\": container with ID starting with b92e37fc9554de98bdd96bd1d923790d05deb5eb56134d67ed14b91683531c6d not found: ID does not exist" Nov 03 20:21:31.933324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.933299 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g"] Nov 03 20:21:31.941534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.941514 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9b65143e-openshift-default-5ddf6c8dd9-lwk8g"] Nov 03 20:21:31.961519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.961482 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:31.985301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.985269 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.985301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.985295 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.985301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.985306 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.985569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.985317 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmzph\" (UniqueName: \"kubernetes.io/projected/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-kube-api-access-bmzph\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:31.985569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:31.985326 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdcdbf31-f878-4515-a5a2-e1b43b1c7420-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:32.647003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:32.646951 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:32.682818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:32.682780 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" path="/var/lib/kubelet/pods/fdcdbf31-f878-4515-a5a2-e1b43b1c7420/volumes" Nov 03 20:21:32.779689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:32.779647 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:33.091209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:33.091170 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:33.887828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:33.887782 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.035653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:34.035617 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:34.599290 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:34.599259 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:35.711186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:35.711148 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:36.626331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:36.626294 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:36.847919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:36.847892 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:21:36.849776 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:36.849751 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:21:36.850984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:36.850963 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:21:36.852807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:36.852784 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:21:37.654638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:37.654597 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:37.745688 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:37.745657 4711 scope.go:122] "RemoveContainer" containerID="80f6d0bbbbd447d72bfe60a374dbaf182484d517697b0f21590e14297cc854d1" Nov 03 20:21:38.416989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:38.416943 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:38.915918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:38.915877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:39.051836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:39.051796 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:39.814363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:39.814327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:40.712771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:40.712710 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:41.264690 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:41.264652 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:41.458804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:41.458742 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:42.292941 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:42.292905 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:42.698042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:42.697941 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:43.001998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:43.001804 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:43.681766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:43.675075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:44.575524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:44.575488 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:44.787963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:44.787911 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:45.929496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:45.929426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:46.140701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:46.140661 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:46.922075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:46.922034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:47.532989 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:21:47.532953 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:21:47.711560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:47.711526 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:48.714226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:48.714187 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:48.941347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:48.941311 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5"] Nov 03 20:21:49.059300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:49.059270 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:49.221615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:49.221570 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:49.355196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:49.355103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:49.505017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:49.504985 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:49.930517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:49.930485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:50.857516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:50.857485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:50.989033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:50.988995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:51.689233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:51.689177 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:52.886121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:52.886076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:53.015393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.015340 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:53.298939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.298905 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58"] Nov 03 20:21:53.299648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.299626 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" Nov 03 20:21:53.299648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.299645 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" Nov 03 20:21:53.299851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.299682 4711 cpu_manager.go:414] "containerMap: removing container" podUID="54d2a0d6-60ee-460b-84f2-29f33842cf08" containerName="payload-processing" Nov 03 20:21:53.299851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.299688 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d2a0d6-60ee-460b-84f2-29f33842cf08" containerName="payload-processing" Nov 03 20:21:53.299851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.299794 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" Nov 03 20:21:53.299851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.299804 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d2a0d6-60ee-460b-84f2-29f33842cf08" containerName="payload-processing" Nov 03 20:21:53.303565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.303545 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.311726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.311704 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-efba68-openshift-default-dockercfg-qvrnj\"" Nov 03 20:21:53.313753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.313556 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-token\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.313753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.313600 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-socket\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.313753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.313637 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-data\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.313753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.313661 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-podinfo\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.314228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.314201 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-credential-socket\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.314336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.314296 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-certs\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.314408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.314389 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-envoy\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.314508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.314461 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istiod-ca-cert\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.314609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.314515 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5b66\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-kube-api-access-j5b66\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.320863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.320835 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58"] Nov 03 20:21:53.380123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.380090 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:53.415761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.415723 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-certs\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.416532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.416506 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-envoy\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.417119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.417100 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istiod-ca-cert\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.418018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.417999 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5b66\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-kube-api-access-j5b66\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.418587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.418567 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-token\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.418998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.418977 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-socket\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.419425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.419407 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-data\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.419888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.419869 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-podinfo\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.421436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.421416 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-credential-socket\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.422289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.417899 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istiod-ca-cert\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.422401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.419832 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-data\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.422510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.419345 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-socket\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.422599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.416362 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-certs\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.422674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.422120 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-credential-socket\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.460579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.459751 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-envoy\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.465526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.465458 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5b66\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-kube-api-access-j5b66\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.487567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.486026 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-token\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.513468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.501075 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-podinfo\") pod \"e2e-ipp-b-efba68-openshift-default-754b857845-ctf58\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.623417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.623338 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:53.879834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.879734 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:53.914692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.914666 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58"] Nov 03 20:21:53.915044 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:21:53.914974 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaee873_ec05_4a07_8f83_6bd0022d10e9.slice/crio-b61d2427f827d7b76af2dc22e8852906bd0a5162fd0cd6741abb50bd9efbdea0 WatchSource:0}: Error finding container b61d2427f827d7b76af2dc22e8852906bd0a5162fd0cd6741abb50bd9efbdea0: Status 404 returned error can't find the container with id b61d2427f827d7b76af2dc22e8852906bd0a5162fd0cd6741abb50bd9efbdea0 Nov 03 20:21:53.917733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.917700 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:53.917837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.917764 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:53.917837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.917794 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:53.941807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:53.941767 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" podUID="3e3b3d29-df89-41e0-8fbc-fe45ad54a366" containerName="payload-processing" containerID="cri-o://1cc8d20a565f9e534e909291056e64bdb26da0de522d638b24b6bf8fa69a42c5" gracePeriod=25 Nov 03 20:21:54.014815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.014765 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:54.135790 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.135753 4711 generic.go:356] "Generic (PLEG): container finished" podID="3e3b3d29-df89-41e0-8fbc-fe45ad54a366" containerID="1cc8d20a565f9e534e909291056e64bdb26da0de522d638b24b6bf8fa69a42c5" exitCode=0 Nov 03 20:21:54.135961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.135848 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" event={"ID":"3e3b3d29-df89-41e0-8fbc-fe45ad54a366","Type":"ContainerDied","Data":"1cc8d20a565f9e534e909291056e64bdb26da0de522d638b24b6bf8fa69a42c5"} Nov 03 20:21:54.139462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.139412 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" event={"ID":"1aaee873-ec05-4a07-8f83-6bd0022d10e9","Type":"ContainerStarted","Data":"b61d2427f827d7b76af2dc22e8852906bd0a5162fd0cd6741abb50bd9efbdea0"} Nov 03 20:21:54.319861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.319829 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:21:54.348121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.348089 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-kube-api-access-xjxml\" (UniqueName: \"kubernetes.io/projected/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-kube-api-access-xjxml\") pod \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\" (UID: \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\") " Nov 03 20:21:54.348319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.348200 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-plugins-config-volume\") pod \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\" (UID: \"3e3b3d29-df89-41e0-8fbc-fe45ad54a366\") " Nov 03 20:21:54.349218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.349185 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-plugins-config-volume" pod "3e3b3d29-df89-41e0-8fbc-fe45ad54a366" (UID: "3e3b3d29-df89-41e0-8fbc-fe45ad54a366"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:54.350986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.350959 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-kube-api-access-xjxml" pod "3e3b3d29-df89-41e0-8fbc-fe45ad54a366" (UID: "3e3b3d29-df89-41e0-8fbc-fe45ad54a366"). InnerVolumeSpecName "kube-api-access-xjxml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:54.407231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.407133 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:54.451535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.451497 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjxml\" (UniqueName: \"kubernetes.io/projected/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-kube-api-access-xjxml\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:54.451682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:54.451536 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3b3d29-df89-41e0-8fbc-fe45ad54a366-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:55.147834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.147783 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" event={"ID":"3e3b3d29-df89-41e0-8fbc-fe45ad54a366","Type":"ContainerDied","Data":"ea1f1d419cead4436323b93ece47b65302f9c5f7932968932daf28aa70dff1ae"} Nov 03 20:21:55.148290 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.147882 4711 scope.go:122] "RemoveContainer" containerID="1cc8d20a565f9e534e909291056e64bdb26da0de522d638b24b6bf8fa69a42c5" Nov 03 20:21:55.148290 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.147794 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5" Nov 03 20:21:55.151439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.151406 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" event={"ID":"1aaee873-ec05-4a07-8f83-6bd0022d10e9","Type":"ContainerStarted","Data":"6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0"} Nov 03 20:21:55.171645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.171618 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5"] Nov 03 20:21:55.174892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.174872 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2218a-669947bdf4-4hqc5"] Nov 03 20:21:55.194340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.194292 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" podStartSLOduration=2.194277927 podStartE2EDuration="2.194277927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:55.191423084 +0000 UTC m=+1218.641529132" watchObservedRunningTime="2026-11-03 20:21:55.194277927 +0000 UTC m=+1218.644383971" Nov 03 20:21:55.209408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.209378 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:55.623858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.623822 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:55.629461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:55.629427 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:56.160470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:56.160428 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:56.161377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:56.161358 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:21:56.164609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:56.164580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:56.300097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:56.300067 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:56.544117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:56.544076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:56.681761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:56.681731 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3b3d29-df89-41e0-8fbc-fe45ad54a366" path="/var/lib/kubelet/pods/3e3b3d29-df89-41e0-8fbc-fe45ad54a366/volumes" Nov 03 20:21:56.715699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:56.715665 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:56.975244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:56.975169 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5"] Nov 03 20:21:56.975426 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:56.975362 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" podUID="6fa0a87a-b607-4444-8022-6c5db411a74e" containerName="maas-api" containerID="cri-o://b2446e1b5bfa9d156f7c065ae6fdc7d3da744c70f654bc063c4fc493a7902cb5" gracePeriod=30 Nov 03 20:21:57.049620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.049585 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:57.172625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.172596 4711 generic.go:356] "Generic (PLEG): container finished" podID="6fa0a87a-b607-4444-8022-6c5db411a74e" containerID="b2446e1b5bfa9d156f7c065ae6fdc7d3da744c70f654bc063c4fc493a7902cb5" exitCode=0 Nov 03 20:21:57.173640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.173612 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" event={"ID":"6fa0a87a-b607-4444-8022-6c5db411a74e","Type":"ContainerDied","Data":"b2446e1b5bfa9d156f7c065ae6fdc7d3da744c70f654bc063c4fc493a7902cb5"} Nov 03 20:21:57.184051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.184026 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:57.290256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.290230 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:21:57.320481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.320434 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:57.402417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.402366 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-metrics-certs\") pod \"6fa0a87a-b607-4444-8022-6c5db411a74e\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " Nov 03 20:21:57.403385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.403360 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-maas-api-tls\") pod \"6fa0a87a-b607-4444-8022-6c5db411a74e\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " Nov 03 20:21:57.403853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.403830 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa0a87a-b607-4444-8022-6c5db411a74e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa0a87a-b607-4444-8022-6c5db411a74e-openshift-service-ca\") pod \"6fa0a87a-b607-4444-8022-6c5db411a74e\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " Nov 03 20:21:57.404510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.404388 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa0a87a-b607-4444-8022-6c5db411a74e-openshift-service-ca" pod "6fa0a87a-b607-4444-8022-6c5db411a74e" (UID: "6fa0a87a-b607-4444-8022-6c5db411a74e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:57.406242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.406220 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa0a87a-b607-4444-8022-6c5db411a74e-kube-api-access-hzrcb\" (UniqueName: \"kubernetes.io/projected/6fa0a87a-b607-4444-8022-6c5db411a74e-kube-api-access-hzrcb\") pod \"6fa0a87a-b607-4444-8022-6c5db411a74e\" (UID: \"6fa0a87a-b607-4444-8022-6c5db411a74e\") " Nov 03 20:21:57.408948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.408925 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa0a87a-b607-4444-8022-6c5db411a74e-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:57.429727 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.427964 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-maas-api-tls" pod "6fa0a87a-b607-4444-8022-6c5db411a74e" (UID: "6fa0a87a-b607-4444-8022-6c5db411a74e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:57.430299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.430227 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa0a87a-b607-4444-8022-6c5db411a74e-kube-api-access-hzrcb" pod "6fa0a87a-b607-4444-8022-6c5db411a74e" (UID: "6fa0a87a-b607-4444-8022-6c5db411a74e"). InnerVolumeSpecName "kube-api-access-hzrcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:57.432143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.432088 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-metrics-certs" pod "6fa0a87a-b607-4444-8022-6c5db411a74e" (UID: "6fa0a87a-b607-4444-8022-6c5db411a74e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:57.510582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.510494 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzrcb\" (UniqueName: \"kubernetes.io/projected/6fa0a87a-b607-4444-8022-6c5db411a74e-kube-api-access-hzrcb\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:57.510582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.510525 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:57.510582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.510534 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa0a87a-b607-4444-8022-6c5db411a74e-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:57.624475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.624392 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:57.767627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.767534 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:57.961467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:57.959008 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:58.181055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:58.180974 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" Nov 03 20:21:58.181055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:58.181004 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5" event={"ID":"6fa0a87a-b607-4444-8022-6c5db411a74e","Type":"ContainerDied","Data":"a1f6003ce76ec93572b136d5d13e0644823f809d6aba2462fe4a4d7f0fbeaab6"} Nov 03 20:21:58.181527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:58.181063 4711 scope.go:122] "RemoveContainer" containerID="b2446e1b5bfa9d156f7c065ae6fdc7d3da744c70f654bc063c4fc493a7902cb5" Nov 03 20:21:58.216715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:58.216683 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5"] Nov 03 20:21:58.220022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:58.220003 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1b7b5e-5bc54c4b5b-zg8p5"] Nov 03 20:21:58.681461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:58.681417 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa0a87a-b607-4444-8022-6c5db411a74e" path="/var/lib/kubelet/pods/6fa0a87a-b607-4444-8022-6c5db411a74e/volumes" Nov 03 20:21:59.019810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:59.019772 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:59.173037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:59.172997 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:21:59.602239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:21:59.602200 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:00.724133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:00.724089 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:01.402946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.402911 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48"] Nov 03 20:22:01.403290 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.403147 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" podUID="02d20df3-65b8-4cd0-bf6c-62d540e85716" containerName="maas-api" containerID="cri-o://cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21" gracePeriod=30 Nov 03 20:22:01.477468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.476282 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:01.607208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.607172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:01.833335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.833309 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:22:01.909413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.909382 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d20df3-65b8-4cd0-bf6c-62d540e85716-kube-api-access-wbgnt\" (UniqueName: \"kubernetes.io/projected/02d20df3-65b8-4cd0-bf6c-62d540e85716-kube-api-access-wbgnt\") pod \"02d20df3-65b8-4cd0-bf6c-62d540e85716\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " Nov 03 20:22:01.909665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.909522 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d20df3-65b8-4cd0-bf6c-62d540e85716-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d20df3-65b8-4cd0-bf6c-62d540e85716-openshift-service-ca\") pod \"02d20df3-65b8-4cd0-bf6c-62d540e85716\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " Nov 03 20:22:01.909665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.909639 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls\") pod \"02d20df3-65b8-4cd0-bf6c-62d540e85716\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " Nov 03 20:22:01.909819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.909780 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-metrics-certs\") pod \"02d20df3-65b8-4cd0-bf6c-62d540e85716\" (UID: \"02d20df3-65b8-4cd0-bf6c-62d540e85716\") " Nov 03 20:22:01.909972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.909935 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d20df3-65b8-4cd0-bf6c-62d540e85716-openshift-service-ca" pod "02d20df3-65b8-4cd0-bf6c-62d540e85716" (UID: "02d20df3-65b8-4cd0-bf6c-62d540e85716"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:01.910360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.910341 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02d20df3-65b8-4cd0-bf6c-62d540e85716-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:01.911883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.911859 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-metrics-certs" pod "02d20df3-65b8-4cd0-bf6c-62d540e85716" (UID: "02d20df3-65b8-4cd0-bf6c-62d540e85716"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:01.911978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.911918 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls" pod "02d20df3-65b8-4cd0-bf6c-62d540e85716" (UID: "02d20df3-65b8-4cd0-bf6c-62d540e85716"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:01.911978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:01.911955 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d20df3-65b8-4cd0-bf6c-62d540e85716-kube-api-access-wbgnt" pod "02d20df3-65b8-4cd0-bf6c-62d540e85716" (UID: "02d20df3-65b8-4cd0-bf6c-62d540e85716"). InnerVolumeSpecName "kube-api-access-wbgnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:02.011680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.011653 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:02.011680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.011678 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbgnt\" (UniqueName: \"kubernetes.io/projected/02d20df3-65b8-4cd0-bf6c-62d540e85716-kube-api-access-wbgnt\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:02.011885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.011688 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02d20df3-65b8-4cd0-bf6c-62d540e85716-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:02.034745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.034714 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:02.218105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.218071 4711 generic.go:356] "Generic (PLEG): container finished" podID="02d20df3-65b8-4cd0-bf6c-62d540e85716" containerID="cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21" exitCode=0 Nov 03 20:22:02.218262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.218133 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" Nov 03 20:22:02.218262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.218155 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" event={"ID":"02d20df3-65b8-4cd0-bf6c-62d540e85716","Type":"ContainerDied","Data":"cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21"} Nov 03 20:22:02.218262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.218237 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48" event={"ID":"02d20df3-65b8-4cd0-bf6c-62d540e85716","Type":"ContainerDied","Data":"5600adabb2be2dd26da4630c762deb2fcc6ab4e8b1054ead63aef6c1444e47cc"} Nov 03 20:22:02.218396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.218267 4711 scope.go:122] "RemoveContainer" containerID="cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21" Nov 03 20:22:02.228600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.228569 4711 scope.go:122] "RemoveContainer" containerID="cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21" Nov 03 20:22:02.228929 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:22:02.228908 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21\": container with ID starting with cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21 not found: ID does not exist" containerID="cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21" Nov 03 20:22:02.229010 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.228939 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21"} err="failed to get container status \"cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21\": rpc error: code = NotFound desc = could not find container \"cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21\": container with ID starting with cb5aa75105ea2ccba4e303c7d57f2038d00ea5a02fb81554fd7a64da8565eb21 not found: ID does not exist" Nov 03 20:22:02.251111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.251081 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48"] Nov 03 20:22:02.255535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.255508 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b9849-5cf8c47c4b-ggq48"] Nov 03 20:22:02.667328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.667296 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:02.680635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.680571 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d20df3-65b8-4cd0-bf6c-62d540e85716" path="/var/lib/kubelet/pods/02d20df3-65b8-4cd0-bf6c-62d540e85716/volumes" Nov 03 20:22:02.831460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.831410 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:02.967322 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:02.967243 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:03.161473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:03.149483 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:03.277077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:03.277041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:03.518089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:03.518058 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.096638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:04.096600 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.246837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:04.246802 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.375333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:04.375243 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:04.715207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:04.715117 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:05.251081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:05.251024 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:06.422317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:06.422285 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:07.426143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:07.426099 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:07.578545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:07.578508 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:07.707339 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:07.707247 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:08.130125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:08.130083 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:08.407628 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:08.407524 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:08.564947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:08.564915 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:09.414223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:09.411239 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:10.244439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:10.244401 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24"] Nov 03 20:22:10.244942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:10.244704 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" containerID="cri-o://62b54c29e2596c5868e66558dc6d3a13f5511aef75930d69169b5cb18b086004" gracePeriod=30 Nov 03 20:22:10.296578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:10.296543 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:10.425552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:10.425514 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:10.561763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:10.561725 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:11.207039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:11.207004 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:11.565156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:11.565121 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:11.987594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:11.987416 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:12.454137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:12.454103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:13.291889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:13.291843 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:13.427979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:13.427947 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:14.135164 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:14.135121 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:14.338219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:14.338181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:15.185972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.185939 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:15.388205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.388174 4711 generic.go:356] "Generic (PLEG): container finished" podID="031104d1-af70-42bc-b158-659f3b3a2f37" containerID="62b54c29e2596c5868e66558dc6d3a13f5511aef75930d69169b5cb18b086004" exitCode=0 Nov 03 20:22:15.388659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.388243 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" event={"ID":"031104d1-af70-42bc-b158-659f3b3a2f37","Type":"ContainerDied","Data":"62b54c29e2596c5868e66558dc6d3a13f5511aef75930d69169b5cb18b086004"} Nov 03 20:22:15.524503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.524469 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:22:15.690999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.690965 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-certs\") pod \"031104d1-af70-42bc-b158-659f3b3a2f37\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " Nov 03 20:22:15.691211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.691083 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-socket\") pod \"031104d1-af70-42bc-b158-659f3b3a2f37\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " Nov 03 20:22:15.691276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.691237 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/031104d1-af70-42bc-b158-659f3b3a2f37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031104d1-af70-42bc-b158-659f3b3a2f37-istio-podinfo\") pod \"031104d1-af70-42bc-b158-659f3b3a2f37\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " Nov 03 20:22:15.691330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.691286 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-istio-token\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-istio-token\") pod \"031104d1-af70-42bc-b158-659f3b3a2f37\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " Nov 03 20:22:15.691384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.691345 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-credential-socket\") pod \"031104d1-af70-42bc-b158-659f3b3a2f37\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " Nov 03 20:22:15.691384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.691359 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-certs" pod "031104d1-af70-42bc-b158-659f3b3a2f37" (UID: "031104d1-af70-42bc-b158-659f3b3a2f37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.691384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.691378 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-kube-api-access-j8nmg\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-kube-api-access-j8nmg\") pod \"031104d1-af70-42bc-b158-659f3b3a2f37\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " Nov 03 20:22:15.691599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.691432 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031104d1-af70-42bc-b158-659f3b3a2f37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031104d1-af70-42bc-b158-659f3b3a2f37-istiod-ca-cert\") pod \"031104d1-af70-42bc-b158-659f3b3a2f37\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " Nov 03 20:22:15.691599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.691505 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-envoy\") pod \"031104d1-af70-42bc-b158-659f3b3a2f37\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " Nov 03 20:22:15.691599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.691586 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-data\") pod \"031104d1-af70-42bc-b158-659f3b3a2f37\" (UID: \"031104d1-af70-42bc-b158-659f3b3a2f37\") " Nov 03 20:22:15.692295 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.692266 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.692547 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.692519 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-data" pod "031104d1-af70-42bc-b158-659f3b3a2f37" (UID: "031104d1-af70-42bc-b158-659f3b3a2f37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.693485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.693429 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-credential-socket" pod "031104d1-af70-42bc-b158-659f3b3a2f37" (UID: "031104d1-af70-42bc-b158-659f3b3a2f37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.693750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.693728 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-socket" pod "031104d1-af70-42bc-b158-659f3b3a2f37" (UID: "031104d1-af70-42bc-b158-659f3b3a2f37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.694237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.694215 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/031104d1-af70-42bc-b158-659f3b3a2f37-istio-podinfo" pod "031104d1-af70-42bc-b158-659f3b3a2f37" (UID: "031104d1-af70-42bc-b158-659f3b3a2f37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:15.694801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.694776 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031104d1-af70-42bc-b158-659f3b3a2f37-istiod-ca-cert" pod "031104d1-af70-42bc-b158-659f3b3a2f37" (UID: "031104d1-af70-42bc-b158-659f3b3a2f37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:15.695336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.695309 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-istio-token" pod "031104d1-af70-42bc-b158-659f3b3a2f37" (UID: "031104d1-af70-42bc-b158-659f3b3a2f37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:15.695657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.695639 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-kube-api-access-j8nmg" pod "031104d1-af70-42bc-b158-659f3b3a2f37" (UID: "031104d1-af70-42bc-b158-659f3b3a2f37"). InnerVolumeSpecName "kube-api-access-j8nmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:15.696890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.696873 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-envoy" pod "031104d1-af70-42bc-b158-659f3b3a2f37" (UID: "031104d1-af70-42bc-b158-659f3b3a2f37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:15.800516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.795910 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031104d1-af70-42bc-b158-659f3b3a2f37-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.800516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.795970 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.800516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.795987 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.800516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.796002 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8nmg\" (UniqueName: \"kubernetes.io/projected/031104d1-af70-42bc-b158-659f3b3a2f37-kube-api-access-j8nmg\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.800516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.796024 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031104d1-af70-42bc-b158-659f3b3a2f37-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.800516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.796038 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.800516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.796052 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.800516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.796066 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031104d1-af70-42bc-b158-659f3b3a2f37-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:15.938303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:15.938266 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:16.397074 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:16.397042 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" Nov 03 20:22:16.397074 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:16.397063 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24" event={"ID":"031104d1-af70-42bc-b158-659f3b3a2f37","Type":"ContainerDied","Data":"70596605e92ea2d0c36002d2b04a6a7cad86dd40fadab8239ca896c6f6ff14a9"} Nov 03 20:22:16.397631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:16.397148 4711 scope.go:122] "RemoveContainer" containerID="62b54c29e2596c5868e66558dc6d3a13f5511aef75930d69169b5cb18b086004" Nov 03 20:22:16.424336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:16.424294 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24"] Nov 03 20:22:16.427677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:16.427638 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-gmn24"] Nov 03 20:22:16.591037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:16.591001 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:16.684723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:16.684623 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" path="/var/lib/kubelet/pods/031104d1-af70-42bc-b158-659f3b3a2f37/volumes" Nov 03 20:22:17.330624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:17.330589 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:17.535622 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:22:17.535579 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:22:18.324317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:18.324275 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.051912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:19.051873 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.178957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:19.178923 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.379799 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:19.379700 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.523712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:19.523678 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:19.930498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:19.930423 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:20.500480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:20.500433 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:21.156991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:21.156945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:21.549441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:21.549407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:21.771858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:21.771809 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:22.263459 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:22.263410 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:22.405468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:22.404436 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:22.820234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:22.820194 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:23.096132 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.096041 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn"] Nov 03 20:22:23.096389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.096333 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" podUID="761300fa-3d54-4401-b409-11657e9ea609" containerName="maas-api" containerID="cri-o://5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24" gracePeriod=30 Nov 03 20:22:23.159595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.159557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:23.290485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.286830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:23.405529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.399404 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:22:23.435086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.434488 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761300fa-3d54-4401-b409-11657e9ea609-kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/761300fa-3d54-4401-b409-11657e9ea609-kube-api-access-l6k8p\") pod \"761300fa-3d54-4401-b409-11657e9ea609\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " Nov 03 20:22:23.435086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.434896 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/761300fa-3d54-4401-b409-11657e9ea609-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761300fa-3d54-4401-b409-11657e9ea609-openshift-service-ca\") pod \"761300fa-3d54-4401-b409-11657e9ea609\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " Nov 03 20:22:23.435086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.434932 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-metrics-certs\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-metrics-certs\") pod \"761300fa-3d54-4401-b409-11657e9ea609\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " Nov 03 20:22:23.435086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.434974 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-maas-api-tls\") pod \"761300fa-3d54-4401-b409-11657e9ea609\" (UID: \"761300fa-3d54-4401-b409-11657e9ea609\") " Nov 03 20:22:23.435519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.435190 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761300fa-3d54-4401-b409-11657e9ea609-kube-api-access-l6k8p" pod "761300fa-3d54-4401-b409-11657e9ea609" (UID: "761300fa-3d54-4401-b409-11657e9ea609"). InnerVolumeSpecName "kube-api-access-l6k8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:23.447494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.435687 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761300fa-3d54-4401-b409-11657e9ea609-openshift-service-ca" pod "761300fa-3d54-4401-b409-11657e9ea609" (UID: "761300fa-3d54-4401-b409-11657e9ea609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:23.447494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.435965 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6k8p\" (UniqueName: \"kubernetes.io/projected/761300fa-3d54-4401-b409-11657e9ea609-kube-api-access-l6k8p\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:23.447494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.435985 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/761300fa-3d54-4401-b409-11657e9ea609-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:23.471544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.465224 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-maas-api-tls" pod "761300fa-3d54-4401-b409-11657e9ea609" (UID: "761300fa-3d54-4401-b409-11657e9ea609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:23.485682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.485572 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-metrics-certs" pod "761300fa-3d54-4401-b409-11657e9ea609" (UID: "761300fa-3d54-4401-b409-11657e9ea609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:23.486198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.486093 4711 generic.go:356] "Generic (PLEG): container finished" podID="761300fa-3d54-4401-b409-11657e9ea609" containerID="5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24" exitCode=0 Nov 03 20:22:23.486198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.486161 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" event={"ID":"761300fa-3d54-4401-b409-11657e9ea609","Type":"ContainerDied","Data":"5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24"} Nov 03 20:22:23.486364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.486212 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" event={"ID":"761300fa-3d54-4401-b409-11657e9ea609","Type":"ContainerDied","Data":"e031328be8df937e9ccdf921bf8d4acf56d403bf4c4e1245574cf619fb0748c9"} Nov 03 20:22:23.486364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.486241 4711 scope.go:122] "RemoveContainer" containerID="5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24" Nov 03 20:22:23.486490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.486383 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn" Nov 03 20:22:23.538297 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.538257 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:23.538546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.538530 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/761300fa-3d54-4401-b409-11657e9ea609-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:23.575410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.575340 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn"] Nov 03 20:22:23.579965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.579919 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d6789f-6c5c9c88df-zmqvn"] Nov 03 20:22:23.584551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.584106 4711 scope.go:122] "RemoveContainer" containerID="5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24" Nov 03 20:22:23.586201 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:22:23.586093 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24\": container with ID starting with 5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24 not found: ID does not exist" containerID="5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24" Nov 03 20:22:23.586201 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.586132 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24"} err="failed to get container status \"5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24\": rpc error: code = NotFound desc = could not find container \"5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24\": container with ID starting with 5061e352858ba070f264633ea193242709c67555fdcfd8c7e0256c8101c78e24 not found: ID does not exist" Nov 03 20:22:23.817998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:23.817949 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:24.680824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:24.680790 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761300fa-3d54-4401-b409-11657e9ea609" path="/var/lib/kubelet/pods/761300fa-3d54-4401-b409-11657e9ea609/volumes" Nov 03 20:22:24.905817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:24.905780 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:25.884612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:25.884576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:26.435293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:26.435255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:26.640166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:26.640093 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:27.036493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:27.032134 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:27.682008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:27.681827 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:27.822525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:27.822488 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:28.046755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:28.046717 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:28.208590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:28.208552 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:28.787468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:28.787310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:28.933893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:28.933186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:29.327659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:29.327612 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:29.719706 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:29.719629 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:29.861476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:29.861420 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.243938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:30.243811 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.375224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:30.375011 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:30.508395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:30.508360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:31.151910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:31.151877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:31.312894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:31.310364 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:31.591200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:31.591162 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:31.933338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:31.933246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:32.537028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:32.536993 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:33.361810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:33.361775 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:34.451253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:34.451218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:34.591525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:34.590624 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:35.249623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:35.249566 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:35.398301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:35.398264 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:35.851037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:35.850996 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:35.999061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:35.998987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:36.149528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:36.148828 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:36.279587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:36.279546 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:37.354933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:37.354888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:37.507469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:37.504866 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:37.798719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:37.798684 4711 scope.go:122] "RemoveContainer" containerID="78654cb0827dc31393149f4c0e5202f1314fe4fd56547f87f1d41c52bc635a62" Nov 03 20:22:38.289355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.289316 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:38.420495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.420429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:38.704099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.704002 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:38.819604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.819568 4711 generic.go:356] "Generic (PLEG): container finished" podID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerID="6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0" exitCode=0 Nov 03 20:22:38.819862 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.819600 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" event={"ID":"1aaee873-ec05-4a07-8f83-6bd0022d10e9","Type":"ContainerDied","Data":"6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0"} Nov 03 20:22:38.820244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.820223 4711 scope.go:122] "RemoveContainer" containerID="6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0" Nov 03 20:22:38.823695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.822791 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:38.823695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.822865 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:38.823695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.822905 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:38.841886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:38.841851 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:39.330918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:39.330879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:39.549467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:39.549361 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:39.828198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:39.828159 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" event={"ID":"1aaee873-ec05-4a07-8f83-6bd0022d10e9","Type":"ContainerStarted","Data":"69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6"} Nov 03 20:22:40.091560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:40.091415 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:40.624101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:40.624063 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:22:40.625768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:40.625742 4711 patch_prober.go:29] interesting pod/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:22:40.625935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:40.625804 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 20:22:40.710279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:40.710244 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:41.160954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:41.160922 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:22:41.483523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:41.481195 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:41.623894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:41.623859 4711 patch_prober.go:29] interesting pod/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:22:41.624104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:41.623912 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 20:22:41.658529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:41.658494 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:42.117314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:42.117270 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:42.564435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:42.564394 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:42.624062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:42.624025 4711 patch_prober.go:29] interesting pod/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:22:42.624274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:42.624087 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 20:22:42.726857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:42.726819 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:43.413924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:43.413889 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:43.623908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:43.623875 4711 patch_prober.go:29] interesting pod/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:22:43.624070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:43.623951 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 20:22:44.115267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.115233 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww"] Nov 03 20:22:44.116107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116082 4711 cpu_manager.go:414] "containerMap: removing container" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" Nov 03 20:22:44.116194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116110 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" Nov 03 20:22:44.116194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116129 4711 cpu_manager.go:414] "containerMap: removing container" podUID="761300fa-3d54-4401-b409-11657e9ea609" containerName="maas-api" Nov 03 20:22:44.116194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116138 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761300fa-3d54-4401-b409-11657e9ea609" containerName="maas-api" Nov 03 20:22:44.116194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116183 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" Nov 03 20:22:44.116194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116190 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" Nov 03 20:22:44.116492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116246 4711 cpu_manager.go:414] "containerMap: removing container" podUID="02d20df3-65b8-4cd0-bf6c-62d540e85716" containerName="maas-api" Nov 03 20:22:44.116492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116254 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d20df3-65b8-4cd0-bf6c-62d540e85716" containerName="maas-api" Nov 03 20:22:44.116492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116282 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6fa0a87a-b607-4444-8022-6c5db411a74e" containerName="maas-api" Nov 03 20:22:44.116492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116290 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa0a87a-b607-4444-8022-6c5db411a74e" containerName="maas-api" Nov 03 20:22:44.116492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116310 4711 cpu_manager.go:414] "containerMap: removing container" podUID="3e3b3d29-df89-41e0-8fbc-fe45ad54a366" containerName="payload-processing" Nov 03 20:22:44.116492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116318 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3b3d29-df89-41e0-8fbc-fe45ad54a366" containerName="payload-processing" Nov 03 20:22:44.116492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116490 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d20df3-65b8-4cd0-bf6c-62d540e85716" containerName="maas-api" Nov 03 20:22:44.116884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116512 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcdbf31-f878-4515-a5a2-e1b43b1c7420" containerName="istio-proxy" Nov 03 20:22:44.116884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116520 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa0a87a-b607-4444-8022-6c5db411a74e" containerName="maas-api" Nov 03 20:22:44.116884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116530 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3b3d29-df89-41e0-8fbc-fe45ad54a366" containerName="payload-processing" Nov 03 20:22:44.116884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116542 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="031104d1-af70-42bc-b158-659f3b3a2f37" containerName="istio-proxy" Nov 03 20:22:44.116884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.116551 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="761300fa-3d54-4401-b409-11657e9ea609" containerName="maas-api" Nov 03 20:22:44.119267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.119243 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.130148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.130117 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww"] Nov 03 20:22:44.131388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.131363 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1142fd-serving-cert\"" Nov 03 20:22:44.204591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.204554 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:44.281464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.277822 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.281464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.278095 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7r7\" (UniqueName: \"kubernetes.io/projected/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-kube-api-access-sq7r7\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.281464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.278227 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-metrics-certs\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.281464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.278305 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.337288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.337246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:44.380535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.380417 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7r7\" (UniqueName: \"kubernetes.io/projected/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-kube-api-access-sq7r7\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.381227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.381201 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-metrics-certs\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.381962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.381941 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.396920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.383235 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.396920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.385170 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.397845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.397820 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-metrics-certs\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.400344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.400317 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.401148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.401122 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7r7\" (UniqueName: \"kubernetes.io/projected/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-kube-api-access-sq7r7\") pod \"maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.446232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.446199 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:44.492125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.492063 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:44.624159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.624122 4711 patch_prober.go:29] interesting pod/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Nov 03 20:22:44.624355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.624175 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Nov 03 20:22:44.756488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.753612 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww"] Nov 03 20:22:44.756488 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:22:44.756231 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32d7bf5_bf81_4f76_9ecf_288ed8af280b.slice/crio-4727f28dc7f4894a640bd7ca92d7c32f417411623a2fa6f73c0b9bd7a4c56d1f WatchSource:0}: Error finding container 4727f28dc7f4894a640bd7ca92d7c32f417411623a2fa6f73c0b9bd7a4c56d1f: Status 404 returned error can't find the container with id 4727f28dc7f4894a640bd7ca92d7c32f417411623a2fa6f73c0b9bd7a4c56d1f Nov 03 20:22:44.759406 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.759386 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:22:44.940825 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:44.940763 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" event={"ID":"c32d7bf5-bf81-4f76-9ecf-288ed8af280b","Type":"ContainerStarted","Data":"4727f28dc7f4894a640bd7ca92d7c32f417411623a2fa6f73c0b9bd7a4c56d1f"} Nov 03 20:22:45.368183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:45.368034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:45.628388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:45.628360 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:22:45.629360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:45.629340 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:22:45.814485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:45.814412 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:45.949110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:45.949014 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" event={"ID":"c32d7bf5-bf81-4f76-9ecf-288ed8af280b","Type":"ContainerStarted","Data":"e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f"} Nov 03 20:22:45.949110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:45.949090 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:45.965499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:45.965463 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:45.970824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:45.970772 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" podStartSLOduration=1.554539735 podStartE2EDuration="1.97075926s" totalImagesPullingTime="416.219525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:45.969483712 +0000 UTC m=+1269.419589758" watchObservedRunningTime="2026-11-03 20:22:45.97075926 +0000 UTC m=+1269.420865306" Nov 03 20:22:46.713493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:46.713377 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:47.083019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.082978 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:47.268481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.267256 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:47.542393 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:22:47.542353 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:22:47.822856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.822775 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb"] Nov 03 20:22:47.826844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.826816 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.835347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.835301 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3f36ae-serving-cert\"" Nov 03 20:22:47.867904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.867538 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb"] Nov 03 20:22:47.877868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.877834 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-metrics-certs\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.878064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.878044 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.878293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.878270 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b156f9-3ce0-4489-bba5-3888e2515e73-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.878370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.878306 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlcc\" (UniqueName: \"kubernetes.io/projected/a2b156f9-3ce0-4489-bba5-3888e2515e73-kube-api-access-8nlcc\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.946873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.946833 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:47.980221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.980186 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-metrics-certs\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.980403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.980291 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.980469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.980411 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b156f9-3ce0-4489-bba5-3888e2515e73-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.980469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.980438 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlcc\" (UniqueName: \"kubernetes.io/projected/a2b156f9-3ce0-4489-bba5-3888e2515e73-kube-api-access-8nlcc\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.981191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.981162 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b156f9-3ce0-4489-bba5-3888e2515e73-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:47.992615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:47.983191 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-metrics-certs\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:48.003332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:48.001414 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:48.053472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:48.053056 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlcc\" (UniqueName: \"kubernetes.io/projected/a2b156f9-3ce0-4489-bba5-3888e2515e73-kube-api-access-8nlcc\") pod \"maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:48.116208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:48.116119 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:48.177679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:48.177644 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:48.463478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:48.463416 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb"] Nov 03 20:22:48.464325 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:22:48.464296 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b156f9_3ce0_4489_bba5_3888e2515e73.slice/crio-4a2f7ef57cdfd90b389a2306ab6140a1cf451dfeded182cc72e8a245e28759b5 WatchSource:0}: Error finding container 4a2f7ef57cdfd90b389a2306ab6140a1cf451dfeded182cc72e8a245e28759b5: Status 404 returned error can't find the container with id 4a2f7ef57cdfd90b389a2306ab6140a1cf451dfeded182cc72e8a245e28759b5 Nov 03 20:22:48.792431 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:48.792385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:48.971929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:48.971903 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" event={"ID":"a2b156f9-3ce0-4489-bba5-3888e2515e73","Type":"ContainerStarted","Data":"4a2f7ef57cdfd90b389a2306ab6140a1cf451dfeded182cc72e8a245e28759b5"} Nov 03 20:22:49.795715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:49.795651 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:49.980327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:49.980283 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" event={"ID":"a2b156f9-3ce0-4489-bba5-3888e2515e73","Type":"ContainerStarted","Data":"6c097a50e0d1a34b0954f4ba712be51340c5a6b4f52341fc8e5f396da6b48d59"} Nov 03 20:22:49.980781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:49.980509 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:50.022479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:50.022409 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" podStartSLOduration=2.542576217 podStartE2EDuration="3.022395847s" totalImagesPullingTime="479.81963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:50.017761828 +0000 UTC m=+1273.467867873" watchObservedRunningTime="2026-11-03 20:22:50.022395847 +0000 UTC m=+1273.472501892" Nov 03 20:22:50.593701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:50.593663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:50.718383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:50.718348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:51.688362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:51.688324 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:51.960812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:51.960711 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:22:52.499177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:52.499148 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:53.233470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:53.218324 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:54.088121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:54.088089 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:54.794812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:54.794771 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:55.616558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:55.616522 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:56.008952 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:56.008922 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:22:56.301224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:56.301135 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:56.947718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:56.947683 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:57.095850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:57.095817 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:57.910893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:57.910621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:58.095105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:58.095074 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:58.239745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:58.239657 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:58.572950 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:58.572910 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:59.058600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:59.058559 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:59.408542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:59.408460 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:59.585280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:59.585242 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:59.719704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:59.719622 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:22:59.876473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:22:59.873136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:00.815901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:00.815864 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:02.080353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:02.080316 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:02.514086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:02.514037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:02.645599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:02.645557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:03.171210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:03.171172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:03.769726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:03.769692 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:03.907948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:03.907879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:04.284440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:04.284388 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:05.115148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:05.115108 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:05.978306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:05.978269 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:06.107136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:06.107095 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:06.598773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:06.598741 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:06.732674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:06.732634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:07.094029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.093975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:07.227391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.227353 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:07.378553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.378476 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659"] Nov 03 20:23:07.384968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.384936 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.400680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.400654 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-82c116-openshift-default-dockercfg-dgfpt\"" Nov 03 20:23:07.407365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.407339 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659"] Nov 03 20:23:07.411265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.411240 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-envoy\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.411617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.411587 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-data\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.411729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.411687 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-certs\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.411789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.411732 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-token\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.411789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.411781 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-podinfo\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.411893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.411807 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-credential-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.411893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.411884 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trdq\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-kube-api-access-9trdq\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.411998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.411953 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.412049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.412024 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istiod-ca-cert\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.515135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.515080 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-data\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.518276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.518252 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-certs\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.520219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.515948 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-data\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.520881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.518824 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-certs\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.521060 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.521041 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-token\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.521243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.521220 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-podinfo\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.521407 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.521262 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-credential-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.521407 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.521362 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trdq\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-kube-api-access-9trdq\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.521639 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.521488 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.521639 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.521618 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istiod-ca-cert\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.521824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.521682 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-envoy\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.522463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.522424 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-credential-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.522897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.522876 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-socket\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.523360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.523244 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istiod-ca-cert\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.538333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.526373 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-envoy\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.614492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.569892 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-podinfo\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.614492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.588334 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-token\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.614492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.592988 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trdq\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-kube-api-access-9trdq\") pod \"e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.658705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.658623 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:07.702091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.702063 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:07.985343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:07.985246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:08.012376 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:23:08.012342 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9910f2ce_27ce_4d9a_9f86_2db12c2ff86a.slice/crio-a62a4aa62e224471cf99de742bcc32d3e741af2f8da19be9ff64c651b06abf9d WatchSource:0}: Error finding container a62a4aa62e224471cf99de742bcc32d3e741af2f8da19be9ff64c651b06abf9d: Status 404 returned error can't find the container with id a62a4aa62e224471cf99de742bcc32d3e741af2f8da19be9ff64c651b06abf9d Nov 03 20:23:08.012601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:08.012575 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659"] Nov 03 20:23:08.015827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:08.015794 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:08.015932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:08.015861 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:08.015932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:08.015891 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:08.115573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:08.115541 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" event={"ID":"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a","Type":"ContainerStarted","Data":"9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9"} Nov 03 20:23:08.115905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:08.115597 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" event={"ID":"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a","Type":"ContainerStarted","Data":"a62a4aa62e224471cf99de742bcc32d3e741af2f8da19be9ff64c651b06abf9d"} Nov 03 20:23:08.141071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:08.141025 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" podStartSLOduration=1.141008885 podStartE2EDuration="1.141008885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:08.139830051 +0000 UTC m=+1291.589936092" watchObservedRunningTime="2026-11-03 20:23:08.141008885 +0000 UTC m=+1291.591114933" Nov 03 20:23:08.284091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:08.284053 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:08.702391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:08.702303 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:09.197110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:09.197068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:09.704245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:09.704212 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:23:09.704469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:09.704264 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 20:23:09.742796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:09.742760 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:09.863219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:09.863182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:10.538750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:10.538696 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:10.702909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:10.702874 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:23:10.703192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:10.703163 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 20:23:11.703298 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:11.703258 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:23:11.703691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:11.703317 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 20:23:11.928235 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:11.928181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:12.075982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:12.075948 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:12.702700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:12.702663 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:23:12.702889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:12.702735 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 20:23:13.261884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:13.261845 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:13.430510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:13.430282 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:13.574161 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:13.574044 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:13.703558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:13.703520 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:23:13.703988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:13.703953 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 20:23:14.707986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:14.707956 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:14.708377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:14.708356 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:14.708968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:14.708947 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:14.855653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:14.855614 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:14.982300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:14.982218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:15.223265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:15.223230 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:15.421293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:15.421258 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:15.778625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:15.778588 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:16.160941 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:16.160842 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:16.475184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:16.475080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:16.603873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:16.603832 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:16.729868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:16.729781 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:17.378132 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:17.378099 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:17.535914 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:23:17.535875 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:23:17.615965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:17.615928 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:17.898346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:17.898308 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:18.079613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:18.079577 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:18.786216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:18.786171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:19.592309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:19.592271 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:20.299119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:20.299082 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:21.166816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:21.166781 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:21.628642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:21.628603 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:21.767007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:21.766976 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:22.115927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:22.115871 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:22.995274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:22.995238 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:23.127360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:23.127326 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:23.650970 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:23.650932 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:24.029013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:24.028974 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:25.352064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:25.352029 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:26.015172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:26.015129 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:26.151136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:26.151101 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:26.331527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:26.331356 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:27.009067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:27.009032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:28.049515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:28.049480 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:28.183637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:28.183570 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:28.577520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:28.577481 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:29.712078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:29.712041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:30.624493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:30.620995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:30.754089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:30.754039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:31.242472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:31.242040 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:31.386853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:31.386820 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:31.519254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:31.519221 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58"] Nov 03 20:23:31.519730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:31.519700 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" containerID="cri-o://69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6" gracePeriod=30 Nov 03 20:23:31.572150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:31.572117 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:32.365519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:32.365486 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:32.625833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:32.625750 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:32.853612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:32.853576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:33.132065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:33.132017 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:33.560159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:33.560122 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:33.697528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:33.697487 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:33.829499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:33.829378 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:34.583171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:34.583138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:35.697404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:35.697356 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:35.827534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:35.827496 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:36.001341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.001241 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:36.799371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.799299 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:23:36.950708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.947966 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-socket\") pod \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " Nov 03 20:23:36.950708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.948159 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-kube-api-access-j5b66\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-kube-api-access-j5b66\") pod \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " Nov 03 20:23:36.950708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.948294 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-credential-socket\") pod \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " Nov 03 20:23:36.950708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.948329 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-token\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-token\") pod \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " Nov 03 20:23:36.950708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.948389 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-podinfo\") pod \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " Nov 03 20:23:36.950708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.948440 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-data\") pod \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " Nov 03 20:23:36.950708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.948500 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-certs\") pod \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " Nov 03 20:23:36.950708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.948546 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-envoy\") pod \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " Nov 03 20:23:36.950708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.948620 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istiod-ca-cert\") pod \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\" (UID: \"1aaee873-ec05-4a07-8f83-6bd0022d10e9\") " Nov 03 20:23:36.951487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.951459 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istiod-ca-cert" pod "1aaee873-ec05-4a07-8f83-6bd0022d10e9" (UID: "1aaee873-ec05-4a07-8f83-6bd0022d10e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:36.951893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.951873 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-socket" pod "1aaee873-ec05-4a07-8f83-6bd0022d10e9" (UID: "1aaee873-ec05-4a07-8f83-6bd0022d10e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:36.953926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.953752 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-credential-socket" pod "1aaee873-ec05-4a07-8f83-6bd0022d10e9" (UID: "1aaee873-ec05-4a07-8f83-6bd0022d10e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:36.953926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.953765 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-certs" pod "1aaee873-ec05-4a07-8f83-6bd0022d10e9" (UID: "1aaee873-ec05-4a07-8f83-6bd0022d10e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:36.954185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.954139 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-data" pod "1aaee873-ec05-4a07-8f83-6bd0022d10e9" (UID: "1aaee873-ec05-4a07-8f83-6bd0022d10e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:36.960531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.959704 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-token" pod "1aaee873-ec05-4a07-8f83-6bd0022d10e9" (UID: "1aaee873-ec05-4a07-8f83-6bd0022d10e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:36.963002 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.962973 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-envoy" pod "1aaee873-ec05-4a07-8f83-6bd0022d10e9" (UID: "1aaee873-ec05-4a07-8f83-6bd0022d10e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:36.986651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.968123 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-kube-api-access-j5b66" pod "1aaee873-ec05-4a07-8f83-6bd0022d10e9" (UID: "1aaee873-ec05-4a07-8f83-6bd0022d10e9"). InnerVolumeSpecName "kube-api-access-j5b66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:37.002465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:36.996639 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-podinfo" pod "1aaee873-ec05-4a07-8f83-6bd0022d10e9" (UID: "1aaee873-ec05-4a07-8f83-6bd0022d10e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:23:37.046237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.046186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:37.050579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.050519 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5b66\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-kube-api-access-j5b66\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:37.050579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.050549 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:37.050579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.050559 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:37.050579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.050567 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:37.050826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.050585 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:37.050826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.050595 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:37.050826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.050610 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:37.050826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.050621 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaee873-ec05-4a07-8f83-6bd0022d10e9-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:37.050826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.050630 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaee873-ec05-4a07-8f83-6bd0022d10e9-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:37.472634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.472540 4711 generic.go:356] "Generic (PLEG): container finished" podID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerID="69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6" exitCode=0 Nov 03 20:23:37.472828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.472738 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" event={"ID":"1aaee873-ec05-4a07-8f83-6bd0022d10e9","Type":"ContainerDied","Data":"69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6"} Nov 03 20:23:37.472828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.472825 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" event={"ID":"1aaee873-ec05-4a07-8f83-6bd0022d10e9","Type":"ContainerDied","Data":"b61d2427f827d7b76af2dc22e8852906bd0a5162fd0cd6741abb50bd9efbdea0"} Nov 03 20:23:37.472939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.472847 4711 scope.go:122] "RemoveContainer" containerID="69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6" Nov 03 20:23:37.474059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.474023 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58" Nov 03 20:23:37.489050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.489031 4711 scope.go:122] "RemoveContainer" containerID="6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0" Nov 03 20:23:37.501176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.501156 4711 scope.go:122] "RemoveContainer" containerID="69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6" Nov 03 20:23:37.501529 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:23:37.501503 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6\": container with ID starting with 69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6 not found: ID does not exist" containerID="69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6" Nov 03 20:23:37.501667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.501540 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6"} err="failed to get container status \"69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6\": rpc error: code = NotFound desc = could not find container \"69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6\": container with ID starting with 69fd9738ccc0fedc0295018a2133c796abf3183937f60b54724811dc26fa01a6 not found: ID does not exist" Nov 03 20:23:37.501667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.501566 4711 scope.go:122] "RemoveContainer" containerID="6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0" Nov 03 20:23:37.501929 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:23:37.501899 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0\": container with ID starting with 6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0 not found: ID does not exist" containerID="6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0" Nov 03 20:23:37.502049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.501935 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0"} err="failed to get container status \"6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0\": rpc error: code = NotFound desc = could not find container \"6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0\": container with ID starting with 6181040025f9257246bccb45a92c0242e4b77b8fa64df006e01966dcbf668fe0 not found: ID does not exist" Nov 03 20:23:37.503530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.503504 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:37.507467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.507431 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58"] Nov 03 20:23:37.522080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.522054 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-efba68-openshift-default-754b857845-ctf58"] Nov 03 20:23:37.650105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:37.650071 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:38.252478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:38.252386 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:38.682954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:38.682920 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" path="/var/lib/kubelet/pods/1aaee873-ec05-4a07-8f83-6bd0022d10e9/volumes" Nov 03 20:23:38.743237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:38.743210 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:38.882398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:38.879748 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:39.233857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:39.233823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:39.957813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:39.956766 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:40.828203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:40.828159 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:41.605403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:41.605368 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:41.877597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:41.877506 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:42.149891 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:42.149787 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:42.492620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:42.492541 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:42.639783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:42.639740 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:43.423771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:43.423734 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:44.597585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:44.597546 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:45.738887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:45.738741 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:46.680705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:46.680663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:47.326931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:47.326897 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:47.533370 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:23:47.533330 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:23:48.018472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:48.015245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:48.626673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:48.626634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:48.949993 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:48.949902 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:49.260506 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:49.260474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:49.401200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:49.401166 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:49.541038 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:49.540950 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:49.801600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:49.801506 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:50.058819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:50.058735 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659"] Nov 03 20:23:50.059251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:50.059213 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerName="istio-proxy" containerID="cri-o://9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9" gracePeriod=30 Nov 03 20:23:50.134023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:50.133986 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:50.277908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:50.277874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:50.946826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:50.946791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.149782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:51.149746 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.459787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:51.459753 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.596582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:51.596545 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:51.982010 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:51.981975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:52.427285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:52.427248 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:53.211834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:53.211794 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:54.083871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:54.083836 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:54.814570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:54.814530 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:55.341077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.341012 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:55.422974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.422943 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-kube-api-access-9trdq\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-kube-api-access-9trdq\") pod \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " Nov 03 20:23:55.423149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.422990 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-token\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-token\") pod \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " Nov 03 20:23:55.423149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423115 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-podinfo\") pod \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " Nov 03 20:23:55.423272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423157 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-socket\") pod \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " Nov 03 20:23:55.423272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423189 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-envoy\") pod \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " Nov 03 20:23:55.423272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423235 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-data\") pod \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " Nov 03 20:23:55.423478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423458 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-credential-socket\") pod \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " Nov 03 20:23:55.423679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423662 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istiod-ca-cert\") pod \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " Nov 03 20:23:55.423887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423530 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-socket" pod "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" (UID: "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.424036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423738 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-credential-socket" pod "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" (UID: "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.424141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.424075 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-certs" pod "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" (UID: "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.424245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.424128 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-data" pod "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" (UID: "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.424338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423998 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istiod-ca-cert" pod "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" (UID: "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:55.424419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.423879 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-certs\") pod \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\" (UID: \"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a\") " Nov 03 20:23:55.425352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.425330 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-token" pod "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" (UID: "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:55.425929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.425905 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.425929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.425930 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.426098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.425941 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.426098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.425967 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.426098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.425980 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.426098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.425994 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.426098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.426024 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-envoy" pod "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" (UID: "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:55.426937 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.426907 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-podinfo" pod "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" (UID: "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:23:55.426937 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.426920 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-kube-api-access-9trdq" pod "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" (UID: "9910f2ce-27ce-4d9a-9f86-2db12c2ff86a"). InnerVolumeSpecName "kube-api-access-9trdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:55.527392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.527358 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trdq\" (UniqueName: \"kubernetes.io/projected/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-kube-api-access-9trdq\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.527392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.527389 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.527591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.527406 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:55.618681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.618578 4711 generic.go:356] "Generic (PLEG): container finished" podID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerID="9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9" exitCode=0 Nov 03 20:23:55.618887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.618705 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" Nov 03 20:23:55.618887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.618707 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" event={"ID":"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a","Type":"ContainerDied","Data":"9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9"} Nov 03 20:23:55.618887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.618831 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659" event={"ID":"9910f2ce-27ce-4d9a-9f86-2db12c2ff86a","Type":"ContainerDied","Data":"a62a4aa62e224471cf99de742bcc32d3e741af2f8da19be9ff64c651b06abf9d"} Nov 03 20:23:55.618887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.618860 4711 scope.go:122] "RemoveContainer" containerID="9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9" Nov 03 20:23:55.635154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.635126 4711 scope.go:122] "RemoveContainer" containerID="9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9" Nov 03 20:23:55.635597 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:23:55.635495 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9\": container with ID starting with 9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9 not found: ID does not exist" containerID="9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9" Nov 03 20:23:55.635597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.635552 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9"} err="failed to get container status \"9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9\": rpc error: code = NotFound desc = could not find container \"9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9\": container with ID starting with 9e3966d6cf080550a421663ba338c7baf48707010c8bb9c57c5444faa66d03c9 not found: ID does not exist" Nov 03 20:23:55.663223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.663194 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659"] Nov 03 20:23:55.669096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.669070 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-82c116-openshift-default-57f8d74ccc-r9659"] Nov 03 20:23:55.675278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.675241 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:55.823166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:55.823132 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.025938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:56.025909 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.334329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:56.334232 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.481521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:56.476112 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:56.683055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:56.682984 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" path="/var/lib/kubelet/pods/9910f2ce-27ce-4d9a-9f86-2db12c2ff86a/volumes" Nov 03 20:23:57.151047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:57.151011 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:57.728751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:57.728710 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:57.895049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:57.895010 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:58.610022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:58.609969 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:58.740310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:58.737377 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:59.410082 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.410037 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb"] Nov 03 20:23:59.410346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.410309 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" podUID="a2b156f9-3ce0-4489-bba5-3888e2515e73" containerName="maas-api" containerID="cri-o://6c097a50e0d1a34b0954f4ba712be51340c5a6b4f52341fc8e5f396da6b48d59" gracePeriod=30 Nov 03 20:23:59.507612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.507580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:59.652582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.652548 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:23:59.661292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.661221 4711 generic.go:356] "Generic (PLEG): container finished" podID="a2b156f9-3ce0-4489-bba5-3888e2515e73" containerID="6c097a50e0d1a34b0954f4ba712be51340c5a6b4f52341fc8e5f396da6b48d59" exitCode=0 Nov 03 20:23:59.661410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.661386 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" event={"ID":"a2b156f9-3ce0-4489-bba5-3888e2515e73","Type":"ContainerDied","Data":"6c097a50e0d1a34b0954f4ba712be51340c5a6b4f52341fc8e5f396da6b48d59"} Nov 03 20:23:59.706216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.706191 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:23:59.805571 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.805540 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b156f9-3ce0-4489-bba5-3888e2515e73-kube-api-access-8nlcc\" (UniqueName: \"kubernetes.io/projected/a2b156f9-3ce0-4489-bba5-3888e2515e73-kube-api-access-8nlcc\") pod \"a2b156f9-3ce0-4489-bba5-3888e2515e73\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " Nov 03 20:23:59.805752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.805740 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b156f9-3ce0-4489-bba5-3888e2515e73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b156f9-3ce0-4489-bba5-3888e2515e73-openshift-service-ca\") pod \"a2b156f9-3ce0-4489-bba5-3888e2515e73\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " Nov 03 20:23:59.805809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.805769 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-metrics-certs\") pod \"a2b156f9-3ce0-4489-bba5-3888e2515e73\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " Nov 03 20:23:59.805869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.805855 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-maas-api-tls\") pod \"a2b156f9-3ce0-4489-bba5-3888e2515e73\" (UID: \"a2b156f9-3ce0-4489-bba5-3888e2515e73\") " Nov 03 20:23:59.806140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.806108 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b156f9-3ce0-4489-bba5-3888e2515e73-openshift-service-ca" pod "a2b156f9-3ce0-4489-bba5-3888e2515e73" (UID: "a2b156f9-3ce0-4489-bba5-3888e2515e73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:59.806553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.806532 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b156f9-3ce0-4489-bba5-3888e2515e73-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:59.807725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.807703 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b156f9-3ce0-4489-bba5-3888e2515e73-kube-api-access-8nlcc" pod "a2b156f9-3ce0-4489-bba5-3888e2515e73" (UID: "a2b156f9-3ce0-4489-bba5-3888e2515e73"). InnerVolumeSpecName "kube-api-access-8nlcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:59.807979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.807957 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-metrics-certs" pod "a2b156f9-3ce0-4489-bba5-3888e2515e73" (UID: "a2b156f9-3ce0-4489-bba5-3888e2515e73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:59.808080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.808019 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-maas-api-tls" pod "a2b156f9-3ce0-4489-bba5-3888e2515e73" (UID: "a2b156f9-3ce0-4489-bba5-3888e2515e73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:59.917530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.914752 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:59.917530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.914790 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2b156f9-3ce0-4489-bba5-3888e2515e73-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:59.917530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:23:59.914808 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nlcc\" (UniqueName: \"kubernetes.io/projected/a2b156f9-3ce0-4489-bba5-3888e2515e73-kube-api-access-8nlcc\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:00.180473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:00.180375 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:00.668714 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:00.668680 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" event={"ID":"a2b156f9-3ce0-4489-bba5-3888e2515e73","Type":"ContainerDied","Data":"4a2f7ef57cdfd90b389a2306ab6140a1cf451dfeded182cc72e8a245e28759b5"} Nov 03 20:24:00.669145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:00.668744 4711 scope.go:122] "RemoveContainer" containerID="6c097a50e0d1a34b0954f4ba712be51340c5a6b4f52341fc8e5f396da6b48d59" Nov 03 20:24:00.669145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:00.668688 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb" Nov 03 20:24:00.694096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:00.694069 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb"] Nov 03 20:24:00.698815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:00.698794 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3f36ae-6c9dc986c6-krrzb"] Nov 03 20:24:01.052556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:01.052523 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:01.186409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:01.186375 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:01.343523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:01.342478 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:02.400187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:02.400148 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:02.538587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:02.538552 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:02.667177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:02.667094 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:02.680344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:02.680316 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2b156f9-3ce0-4489-bba5-3888e2515e73" path="/var/lib/kubelet/pods/a2b156f9-3ce0-4489-bba5-3888e2515e73/volumes" Nov 03 20:24:02.993475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:02.993371 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:03.394702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:03.394669 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:03.560201 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:03.558053 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:04.227516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:04.227473 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:04.702602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:04.701170 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:05.437701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:05.437658 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:05.600469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:05.598001 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:05.927386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:05.927346 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:06.303287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:06.303256 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:06.720841 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:06.720717 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:07.510432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:07.510385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:08.246744 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:08.246702 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:08.892119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:08.892080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:09.168590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:09.168505 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:09.599511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:09.599437 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:09.823623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:09.823580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:12.131568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:12.131525 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:12.257942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:12.257891 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:12.782737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:12.782683 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:12.918854 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:12.918812 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:13.344575 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:13.344505 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:13.799778 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:13.799742 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:14.132563 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:14.132439 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:14.790692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:14.790653 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:15.363015 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:15.362975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:15.497467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:15.497399 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:15.905800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:15.905753 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:16.051699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:16.051661 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:17.476313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:17.476229 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:17.550874 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:24:17.549984 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:24:17.600320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:17.600282 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:18.340097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:18.340059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:18.534868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:18.534837 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:19.048795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:19.048735 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:19.176942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:19.176893 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:19.334821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:19.334660 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:20.212286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:20.212246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:20.387955 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:20.387705 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:21.112944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:21.112911 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:21.592554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:21.592516 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:22.301979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:22.301932 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:23.205932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:23.205891 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:23.329956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:23.329903 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:23.821269 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:23.821234 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:23.950292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:23.950260 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:24.613376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:24.613335 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:24.746675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:24.746609 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:24.890274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:24.890169 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:25.131612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:25.131576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:25.466624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:25.466588 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:25.922118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:25.922081 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:26.372084 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:26.372037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:27.188011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:27.187970 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:28.002842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:28.002801 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:28.162589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:28.162539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:28.988972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:28.988936 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:29.804991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:29.804956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:30.817267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:30.817231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:31.263979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:31.263943 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:31.395946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:31.395908 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:31.783735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:31.783674 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:31.969824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:31.969779 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:32.205976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:32.205888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:33.011236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:33.011196 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:34.005430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:34.005391 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:34.708028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:34.707995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:35.391493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:35.391421 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:36.111300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:36.111259 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:36.827693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:36.827646 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:37.554757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:37.554721 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:38.624458 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:38.624409 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:39.166526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:39.166488 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:39.382554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:39.382514 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:40.226423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:40.226380 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:40.900239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:40.900198 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:41.721763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:41.721721 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:43.889466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:43.888527 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:44.019889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:44.019857 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:44.594570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:44.594536 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:45.257206 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:45.257171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:46.202165 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:46.202126 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:47.126469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:47.125625 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:47.819497 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:24:47.793580 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:24:48.000892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:48.000854 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:48.352628 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:48.352572 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.370430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:48.361340 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:24:48.425469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:48.424588 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:48.539631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:48.539586 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.713598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:48.705650 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.807066 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:48.807022 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.852774 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:48.852724 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:48.924172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:48.924140 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:49.998521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:49.998478 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:51.057561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:51.057522 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:51.938765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:51.938736 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:52.750150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:52.750117 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:53.346551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:53.346512 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:53.874126 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:53.874092 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:54.668722 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:54.668691 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:55.056901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:55.056867 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:55.188758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:55.188713 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:55.404371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:55.404284 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:55.606311 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:55.606273 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:55.941440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:55.941408 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:56.541727 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:56.541680 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:56.702824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:56.702793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:57.622187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:57.622140 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:58.523274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:58.523240 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:24:59.431263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:24:59.431229 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:00.391625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:00.391480 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:01.315802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:01.315762 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:01.816091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:01.816045 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:02.522010 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:02.521965 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:03.715069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:03.715032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:04.478024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:04.477978 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:04.807570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:04.807536 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:05.727071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:05.727034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:06.665697 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:06.665660 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:07.830624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:07.830573 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:08.068481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:08.066869 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:08.220303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:08.220203 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:08.894465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:08.894400 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:09.788144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:09.788108 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:10.546531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.546498 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r"] Nov 03 20:25:10.547437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547407 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerName="istio-proxy" Nov 03 20:25:10.547437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547433 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerName="istio-proxy" Nov 03 20:25:10.547646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547499 4711 cpu_manager.go:414] "containerMap: removing container" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" Nov 03 20:25:10.547646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547508 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" Nov 03 20:25:10.547646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547520 4711 cpu_manager.go:414] "containerMap: removing container" podUID="a2b156f9-3ce0-4489-bba5-3888e2515e73" containerName="maas-api" Nov 03 20:25:10.547646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547527 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b156f9-3ce0-4489-bba5-3888e2515e73" containerName="maas-api" Nov 03 20:25:10.547646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547565 4711 cpu_manager.go:414] "containerMap: removing container" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" Nov 03 20:25:10.547646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547571 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" Nov 03 20:25:10.547932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547735 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9910f2ce-27ce-4d9a-9f86-2db12c2ff86a" containerName="istio-proxy" Nov 03 20:25:10.547932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547751 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" Nov 03 20:25:10.547932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547760 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b156f9-3ce0-4489-bba5-3888e2515e73" containerName="maas-api" Nov 03 20:25:10.547932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.547777 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aaee873-ec05-4a07-8f83-6bd0022d10e9" containerName="istio-proxy" Nov 03 20:25:10.551552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.551529 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.567562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.567291 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r"] Nov 03 20:25:10.593645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.593610 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.593645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.593649 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.593847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.593765 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.593847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.593807 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.593944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.593867 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.593944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.593893 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.594040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.593943 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmcwg\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-kube-api-access-pmcwg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.594040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.593969 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.594040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.593994 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.618896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.618860 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:10.722466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.718524 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.773223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.773188 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.784468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.773559 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.784785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.784758 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.784986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.784969 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.795573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.795541 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.795813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.795793 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmcwg\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-kube-api-access-pmcwg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.795966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.795952 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.797190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.797110 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.797987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.787064 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-certs\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.798812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.732551 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istiod-ca-cert\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.798936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.798430 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-data\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.799097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.798710 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-credential-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.799210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.786511 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-socket\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.803752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.803700 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-token\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.804814 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.804770 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-envoy\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.805321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.805301 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-podinfo\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.818275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.818209 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmcwg\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-kube-api-access-pmcwg\") pod \"e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:10.875486 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:10.875410 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:11.174860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:11.174832 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r"] Nov 03 20:25:11.176970 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:25:11.176937 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefc927f_55b7_47d4_b69e_e8bb133ec73e.slice/crio-b7386e9ff24f3128321e4206078d1bd5a2bceb64e7cc9c649d375ba955034d90 WatchSource:0}: Error finding container b7386e9ff24f3128321e4206078d1bd5a2bceb64e7cc9c649d375ba955034d90: Status 404 returned error can't find the container with id b7386e9ff24f3128321e4206078d1bd5a2bceb64e7cc9c649d375ba955034d90 Nov 03 20:25:11.179842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:11.179805 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:11.179934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:11.179883 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:11.179934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:11.179914 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:11.799470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:11.798989 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:12.125265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:12.125172 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" event={"ID":"eefc927f-55b7-47d4-b69e-e8bb133ec73e","Type":"ContainerStarted","Data":"9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3"} Nov 03 20:25:12.125265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:12.125247 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" event={"ID":"eefc927f-55b7-47d4-b69e-e8bb133ec73e","Type":"ContainerStarted","Data":"b7386e9ff24f3128321e4206078d1bd5a2bceb64e7cc9c649d375ba955034d90"} Nov 03 20:25:12.152877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:12.152575 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" podStartSLOduration=2.152556357 podStartE2EDuration="2.152556357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:12.151703029 +0000 UTC m=+1415.601809076" watchObservedRunningTime="2026-11-03 20:25:12.152556357 +0000 UTC m=+1415.602662403" Nov 03 20:25:12.592852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:12.592813 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:13.111552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:13.075686 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:25:13.111552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:13.075748 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 20:25:13.164470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:13.147713 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:13.655859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:13.655817 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:13.890470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:13.889892 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:25:13.890470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:13.889959 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 20:25:14.546915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:14.546873 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:14.682674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:14.678857 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:14.890907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:14.890810 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:25:14.891086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:14.890904 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 20:25:14.902796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:14.902768 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:15.215896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:15.215815 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:15.591536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:15.591485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:15.886935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:15.886837 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:25:15.887195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:15.887162 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 20:25:16.402160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:16.402125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:16.881476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:16.876831 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:25:16.881476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:16.876892 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 20:25:17.329980 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:17.329941 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:17.534304 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:25:17.534264 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:25:17.876019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:17.875981 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:25:17.876186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:17.876045 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 03 20:25:18.236299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:18.236190 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:18.916690 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:18.916662 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:18.917208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:18.917181 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:18.918142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:18.918120 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:25:19.295099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:19.295061 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:20.103087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:20.103047 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:21.065748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:21.065712 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:21.818025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:21.817977 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:22.608295 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:22.608250 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:23.319853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:23.319816 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:24.161493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:24.161455 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:24.945736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:24.945693 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:25.717967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:25.717924 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:25.921282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:25.921246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:26.404866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:26.404832 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:27.069616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:27.069582 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:27.804465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:27.796055 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:28.518365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:28.518327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:29.192648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:29.192613 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:30.022684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:30.022652 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:30.708946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:30.708912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:31.072438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:31.072407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:31.692382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:31.692351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:32.516125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:32.515507 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:34.698600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:34.698226 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:25:34.797603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:34.797556 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:34.798934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:34.798810 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.032599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.032547 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.128592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.128542 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.147569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.147526 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.304236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.304136 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.340771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.340716 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.465579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.465531 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:25:35.711693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.711575 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.753472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.728188 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:25:35.819469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.804869 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.819469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.807518 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:35.845471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.822250 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:25:35.845471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.822319 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:25:35.845471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.844873 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 03 20:25:35.845471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:35.844990 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.074468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.045550 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.074468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.045947 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:25:36.132470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.115085 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:36.182510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.157518 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.296582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.296547 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:25:36.326034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.285066 4711 patch_prober.go:29] interesting pod/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:36.326034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.308463 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.326034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.308811 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.442471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.327600 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:25:36.442471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.333551 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Nov 03 20:25:36.442471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.367422 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.442471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.367985 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:25:36.442471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.396430 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Nov 03 20:25:36.474809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.474755 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.489989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.489946 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:25:36.911781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.757160 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:25:36.911781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.849528 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:36.911781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.851285 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:25:36.911781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.884495 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:25:36.987470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:36.987272 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.056471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.055463 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.222078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.221977 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:25:37.311475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.309495 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.311475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.309794 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.379672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.378788 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.490832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.490716 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.735469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.734583 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:37.735469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.734666 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.834530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.834484 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:25:37.836572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.836529 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:37.836706 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.836583 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:37.891467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.865493 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:25:37.891978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.875185 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:25:37.892701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.891084 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.020466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.999318 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:38.020466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:37.999387 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.261691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.237370 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:25:38.341363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.341298 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.387006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.386956 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.498775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.498721 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.507793 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.500301 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:38.578958 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.578871 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:25:38.622466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.615259 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:25:38.622466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.620069 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:25:38.622466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.620142 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:25:38.624281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.624235 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:25:38.634977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.634901 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:25:38.636352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.636332 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:25:38.642314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.642279 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:25:38.649816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:38.649795 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:25:39.004674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:39.004640 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:39.994394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:39.994350 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:40.741046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:40.740872 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:41.696514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:41.687997 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:42.648752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:42.648706 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:43.725356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:43.725317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:44.630382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:44.630344 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:45.408357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:45.408319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:46.186386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:46.186340 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:46.917557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:46.915049 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:47.533596 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:25:47.533560 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:25:49.288245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:49.288206 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:50.303827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:50.303787 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:51.222919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:51.222882 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:52.102463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:52.102408 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:52.853946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:52.853905 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:53.586382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:53.586344 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:54.569470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:54.560414 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:54.907233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:54.907133 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:55.428327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:55.424878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:56.122388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.122263 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:56.367704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.367620 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:56.730471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.719766 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:56.730471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.724527 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:25:56.730471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.724981 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:25:56.730471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.725299 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:25:56.731123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.731093 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:56.854640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.854586 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:25:56.892630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.892576 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:56.918555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.918501 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:56.949468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:56.944659 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:57.320492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.313886 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:57.443477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.443092 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:25:57.496419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.486638 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:25:57.496419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.486693 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:57.504724 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.497732 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:57.774579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.774538 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:25:57.775561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.775537 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:25:57.781892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.779511 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:25:57.782402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.782381 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:25:57.881515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.855333 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:57.919482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:57.882325 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:25:58.311181 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:58.311147 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:25:58.520493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:58.520077 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:25:58.642077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:58.641969 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:25:58.866791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:58.866298 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:25:58.953125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:25:58.953036 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:26:01.041650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:01.041613 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:02.076940 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:02.076904 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:03.139323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:03.139280 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:03.773119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:03.773077 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:03.908422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:03.908384 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:04.504134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:04.504096 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:05.091278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:05.091249 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:05.234308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:05.234207 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:05.817498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:05.817457 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:06.253777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:06.253663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:06.385274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:06.385217 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:06.660291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:06.660256 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:07.601123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:07.601081 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:07.920749 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:07.920666 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:09.053940 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:09.053901 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:09.535917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:09.535874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:09.914992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:09.914903 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:10.815857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:10.815820 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:10.941902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:10.941861 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:11.115004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:11.114907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:11.803292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:11.803239 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:12.804272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:12.804231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:14.326098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:14.326047 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:14.406483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:14.405081 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:26:14.530607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:14.530546 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:14.626391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:14.626284 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:14.627046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:14.627016 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:26:14.650033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:14.649977 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:14.882559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:14.881270 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:14.887580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:14.887532 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.129296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.128939 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 03 20:26:15.288987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.288284 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:15.327796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.327747 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.371502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.370587 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:26:15.426469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.425363 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.511485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.483921 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:26:15.551529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.531125 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:26:15.551529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.531877 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:26:15.629117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.629065 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.687471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.648114 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:26:15.687721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.687683 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:26:15.731637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.689565 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:26:15.731637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.686833 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.731637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.728574 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:26:15.731637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.729962 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:26:15.846019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.845513 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Nov 03 20:26:15.947104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.947059 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:26:15.948074 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.948048 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:26:15.965498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.951979 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:15.965498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:15.952298 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:26:16.070470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.067403 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:16.359625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.359573 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.364583 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.364539 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Nov 03 20:26:16.393470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.368573 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.426714 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.426661 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.533598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.533549 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.648530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.623399 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:16.738211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.695931 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:26:16.750319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.731625 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.877767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.866580 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:16.998070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:16.997956 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:17.117731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.117672 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:17.118669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.118634 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 03 20:26:17.120215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.120186 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:17.356753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.340590 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.374773 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.374822 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.374854 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.374890 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.374917 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.374946 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.374977 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.375007 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.375034 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:26:17.375501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.375062 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:26:17.541845 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:26:17.541804 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:26:17.878582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:17.878544 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:19.025238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:19.025158 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:19.721304 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:19.721248 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:20.796594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:20.796562 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:21.515218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:21.515160 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:21.994275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:21.994178 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:22.295707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:22.295668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:23.273699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:23.273651 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:23.592423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:23.592330 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:23.728975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:23.728938 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:23.902129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:23.902041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:24.199872 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:24.199785 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:24.338044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:24.337979 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:24.791240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:24.791198 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:24.939058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:24.939015 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:25.294336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:25.294287 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:25.512962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:25.512925 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:25.667533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:25.658986 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:26.101496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:26.098379 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:26.860370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:26.860330 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:27.593867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:27.593829 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:27.737073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:27.736062 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:28.193208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:28.191968 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:28.508191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:28.508155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:28.905620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:28.905529 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:29.057317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:29.057279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:29.186641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:29.186559 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:29.323233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:29.323190 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:29.573497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:29.573436 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:29.738944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:29.738910 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:30.173571 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:30.173532 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:30.312091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:30.312057 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:31.308479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:31.308395 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:32.294275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:32.294230 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:32.425297 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:32.425262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:32.551733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:32.551647 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:32.807508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:32.807380 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:32.948187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:32.948141 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:33.096698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:33.096167 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:33.244878 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:33.244842 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:33.408072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:33.407986 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:34.038956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:34.038904 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:34.176604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:34.176570 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:34.676894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:34.676851 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.502761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:35.502731 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:35.874478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:35.874421 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.312623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:36.312589 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.444601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:36.444567 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:36.905826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:36.905799 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:26:36.907960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:36.907939 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:26:36.908982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:36.908961 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:26:36.910843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:36.910823 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:26:36.996085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:36.996049 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww"] Nov 03 20:26:36.996369 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:36.996339 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" podUID="c32d7bf5-bf81-4f76-9ecf-288ed8af280b" containerName="maas-api" containerID="cri-o://e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f" gracePeriod=30 Nov 03 20:26:37.070508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.070426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:37.201665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.201351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:37.364496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.361617 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:26:37.449548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.444269 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-openshift-service-ca\") pod \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " Nov 03 20:26:37.449548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.444416 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-metrics-certs\") pod \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " Nov 03 20:26:37.449548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.444616 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-kube-api-access-sq7r7\" (UniqueName: \"kubernetes.io/projected/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-kube-api-access-sq7r7\") pod \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " Nov 03 20:26:37.449548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.444696 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-maas-api-tls\") pod \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\" (UID: \"c32d7bf5-bf81-4f76-9ecf-288ed8af280b\") " Nov 03 20:26:37.449548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.446060 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-openshift-service-ca" pod "c32d7bf5-bf81-4f76-9ecf-288ed8af280b" (UID: "c32d7bf5-bf81-4f76-9ecf-288ed8af280b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:37.472916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.456043 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-kube-api-access-sq7r7" pod "c32d7bf5-bf81-4f76-9ecf-288ed8af280b" (UID: "c32d7bf5-bf81-4f76-9ecf-288ed8af280b"). InnerVolumeSpecName "kube-api-access-sq7r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:37.472916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.459862 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-metrics-certs" pod "c32d7bf5-bf81-4f76-9ecf-288ed8af280b" (UID: "c32d7bf5-bf81-4f76-9ecf-288ed8af280b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:37.472916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.462092 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:37.477543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.477490 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-maas-api-tls" pod "c32d7bf5-bf81-4f76-9ecf-288ed8af280b" (UID: "c32d7bf5-bf81-4f76-9ecf-288ed8af280b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:26:37.546572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.546535 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:37.546572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.546566 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:37.546572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.546576 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq7r7\" (UniqueName: \"kubernetes.io/projected/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-kube-api-access-sq7r7\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:37.546803 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.546587 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c32d7bf5-bf81-4f76-9ecf-288ed8af280b-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:37.605596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.605466 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:37.629059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.629025 4711 generic.go:356] "Generic (PLEG): container finished" podID="c32d7bf5-bf81-4f76-9ecf-288ed8af280b" containerID="e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f" exitCode=0 Nov 03 20:26:37.629314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.629219 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" event={"ID":"c32d7bf5-bf81-4f76-9ecf-288ed8af280b","Type":"ContainerDied","Data":"e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f"} Nov 03 20:26:37.629314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.629279 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" event={"ID":"c32d7bf5-bf81-4f76-9ecf-288ed8af280b","Type":"ContainerDied","Data":"4727f28dc7f4894a640bd7ca92d7c32f417411623a2fa6f73c0b9bd7a4c56d1f"} Nov 03 20:26:37.629314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.629309 4711 scope.go:122] "RemoveContainer" containerID="e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f" Nov 03 20:26:37.629604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.629584 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww" Nov 03 20:26:37.651700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.651670 4711 scope.go:122] "RemoveContainer" containerID="e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f" Nov 03 20:26:37.652637 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:26:37.652601 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f\": container with ID starting with e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f not found: ID does not exist" containerID="e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f" Nov 03 20:26:37.652811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.652648 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f"} err="failed to get container status \"e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f\": rpc error: code = NotFound desc = could not find container \"e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f\": container with ID starting with e8f54b9acd280f9bab8afa68a072c8f522dcc30f3c5297bb3790f90122f6239f not found: ID does not exist" Nov 03 20:26:37.664651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.664625 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww"] Nov 03 20:26:37.672139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.672113 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1142fd-7b85b9f665-4c2ww"] Nov 03 20:26:37.888654 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:37.888614 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:38.071470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:38.043677 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:38.681751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:38.681716 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c32d7bf5-bf81-4f76-9ecf-288ed8af280b" path="/var/lib/kubelet/pods/c32d7bf5-bf81-4f76-9ecf-288ed8af280b/volumes" Nov 03 20:26:38.830360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:38.830326 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:39.180566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:39.180508 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:39.320471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:39.317391 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:39.822262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:39.822224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:40.594819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:40.594783 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:41.117048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:41.117010 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:41.297910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:41.297874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:41.464965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:41.463339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:41.599949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:41.599912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:42.350698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:42.350663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:43.009629 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:43.009592 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:43.190470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:43.190160 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:44.137516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:44.137474 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:44.166686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:44.166623 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:26:44.293467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:44.292868 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:44.319294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:44.318546 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:26:44.346572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:44.346526 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded" Nov 03 20:26:44.363566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:44.363522 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:44.365020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:44.364987 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:26:44.663472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:44.659676 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:44.767470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:44.762948 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:26:45.288467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.286393 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:26:45.325490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.319323 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:45.493500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.345930 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:26:45.493500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.346614 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 20:26:45.493500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.376866 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:26:45.493500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.413790 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:26:45.513483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.502155 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:26:45.673520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.638921 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:26:45.673520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.639818 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:26:45.717518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.710329 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:26:45.721837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.721806 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:26:45.824048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.823994 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:45.824433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.824410 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:26:45.990354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.990256 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 03 20:26:46.160590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:45.990914 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:46.160961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.050742 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:46.162592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.146260 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:46.163144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.163124 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:26:46.232244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.232171 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:26:46.378343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.378310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:46.417469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.388229 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:26:46.418081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.389777 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:26:46.430406 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.395796 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:26:46.430810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.430793 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:26:46.447884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.447858 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:26:46.471530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.471501 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:26:46.478845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.477959 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:26:46.484194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.484172 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:26:46.866757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:46.866725 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:47.515902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:47.515866 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:47.540680 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:26:47.540645 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:26:48.005033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:48.004995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:48.445863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:48.445823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:48.767751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:48.767716 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:48.898461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:48.898413 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:49.225535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:49.225407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:49.606641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:49.606607 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:49.740620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:49.740584 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:49.875527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:49.872383 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:50.494238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:50.494183 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:51.328198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.328162 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc"] Nov 03 20:26:51.329095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.329071 4711 cpu_manager.go:414] "containerMap: removing container" podUID="c32d7bf5-bf81-4f76-9ecf-288ed8af280b" containerName="maas-api" Nov 03 20:26:51.329095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.329095 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c32d7bf5-bf81-4f76-9ecf-288ed8af280b" containerName="maas-api" Nov 03 20:26:51.329276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.329242 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="c32d7bf5-bf81-4f76-9ecf-288ed8af280b" containerName="maas-api" Nov 03 20:26:51.334630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.334609 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.342951 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.342924 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc"] Nov 03 20:26:51.358981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.358957 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-bab2c3-serving-cert\"" Nov 03 20:26:51.419802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.419764 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:51.426748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.426719 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.426942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.426923 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.427008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.426960 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0289e060-4030-4262-8f32-2d4d656e71c0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.427008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.426991 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzn5\" (UniqueName: \"kubernetes.io/projected/0289e060-4030-4262-8f32-2d4d656e71c0-kube-api-access-gbzn5\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.528689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.528649 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.533705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.533648 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.537772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.537720 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0289e060-4030-4262-8f32-2d4d656e71c0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.538068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.538048 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzn5\" (UniqueName: \"kubernetes.io/projected/0289e060-4030-4262-8f32-2d4d656e71c0-kube-api-access-gbzn5\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.547144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.538527 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0289e060-4030-4262-8f32-2d4d656e71c0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.551622 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.551590 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.551797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.551772 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:51.555335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.555306 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzn5\" (UniqueName: \"kubernetes.io/projected/0289e060-4030-4262-8f32-2d4d656e71c0-kube-api-access-gbzn5\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.556955 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.556936 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.653300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.653211 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:51.840205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.840170 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:51.948321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.948292 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc"] Nov 03 20:26:51.949269 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:26:51.949242 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0289e060_4030_4262_8f32_2d4d656e71c0.slice/crio-3d61b5a78a0fbab03cd945269e88b6455e6e4ec9fbe2e07c2461efbae17a4b83 WatchSource:0}: Error finding container 3d61b5a78a0fbab03cd945269e88b6455e6e4ec9fbe2e07c2461efbae17a4b83: Status 404 returned error can't find the container with id 3d61b5a78a0fbab03cd945269e88b6455e6e4ec9fbe2e07c2461efbae17a4b83 Nov 03 20:26:51.996876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:51.996846 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:52.257397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:52.257357 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" event={"ID":"0289e060-4030-4262-8f32-2d4d656e71c0","Type":"ContainerStarted","Data":"3d61b5a78a0fbab03cd945269e88b6455e6e4ec9fbe2e07c2461efbae17a4b83"} Nov 03 20:26:52.598355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:52.598271 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:53.117805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:53.117757 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:53.272499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:53.263116 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:53.335511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:53.335461 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" event={"ID":"0289e060-4030-4262-8f32-2d4d656e71c0","Type":"ContainerStarted","Data":"5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785"} Nov 03 20:26:53.336908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:53.336884 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:26:53.381522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:53.380721 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" podStartSLOduration=1.992378988 podStartE2EDuration="2.380699401s" totalImagesPullingTime="388.320413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:53.365964607 +0000 UTC m=+1516.816070655" watchObservedRunningTime="2026-11-03 20:26:53.380699401 +0000 UTC m=+1516.830805449" Nov 03 20:26:54.109428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:54.109385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:54.357051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:54.357013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:54.557426 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:54.557391 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:55.101959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:55.101927 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:55.557705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:55.557651 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:55.784026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:55.783986 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:56.212432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:56.212398 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:56.994479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:56.994429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:57.132904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:57.132870 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:57.494418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:57.494367 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:57.954141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:57.954067 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:58.806056 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:58.806020 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:59.179124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:59.179021 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:26:59.540466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:26:59.540114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:00.201037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:00.200995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:00.354398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:00.354366 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 20:27:00.879839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:00.879802 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:01.097626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:01.097589 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:01.888915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:01.888875 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:02.032463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:02.032410 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:02.803091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:02.803060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:02.940347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:02.940313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:03.229906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:03.229819 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:03.762364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:03.762328 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:03.896659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:03.895917 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:04.162489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:04.162373 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:04.880512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:04.880017 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:05.122495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.122456 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8"] Nov 03 20:27:05.127969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.127945 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.136755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.136678 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-30e9de-serving-cert\"" Nov 03 20:27:05.139637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.139614 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8"] Nov 03 20:27:05.206046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.206013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:05.255240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.255214 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72411ecc-c8d4-4366-9838-0670268e3d05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.255404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.255378 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.255506 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.255435 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.255653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.255632 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75kls\" (UniqueName: \"kubernetes.io/projected/72411ecc-c8d4-4366-9838-0670268e3d05-kube-api-access-75kls\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.348796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.348739 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:05.357240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.357209 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.357376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.357302 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.357499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.357439 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kls\" (UniqueName: \"kubernetes.io/projected/72411ecc-c8d4-4366-9838-0670268e3d05-kube-api-access-75kls\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.357614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.357591 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72411ecc-c8d4-4366-9838-0670268e3d05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.358487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.358464 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72411ecc-c8d4-4366-9838-0670268e3d05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.359857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.359834 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.359935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.359915 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.369475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.369437 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75kls\" (UniqueName: \"kubernetes.io/projected/72411ecc-c8d4-4366-9838-0670268e3d05-kube-api-access-75kls\") pod \"maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.460041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.449316 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:05.484085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.484053 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:05.800051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:05.799942 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8"] Nov 03 20:27:05.809617 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:27:05.809583 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72411ecc_c8d4_4366_9838_0670268e3d05.slice/crio-014ab7b503354707bac4e89e3007c3cf03cfe72808b1f843684c7e998cec35cb WatchSource:0}: Error finding container 014ab7b503354707bac4e89e3007c3cf03cfe72808b1f843684c7e998cec35cb: Status 404 returned error can't find the container with id 014ab7b503354707bac4e89e3007c3cf03cfe72808b1f843684c7e998cec35cb Nov 03 20:27:06.204233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:06.204152 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:06.463470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:06.463377 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" event={"ID":"72411ecc-c8d4-4366-9838-0670268e3d05","Type":"ContainerStarted","Data":"b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23"} Nov 03 20:27:06.463470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:06.463427 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" event={"ID":"72411ecc-c8d4-4366-9838-0670268e3d05","Type":"ContainerStarted","Data":"014ab7b503354707bac4e89e3007c3cf03cfe72808b1f843684c7e998cec35cb"} Nov 03 20:27:06.463873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:06.463489 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:06.493250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:06.493197 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" podStartSLOduration=1.008835806 podStartE2EDuration="1.493182503s" totalImagesPullingTime="484.346697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:06.488860122 +0000 UTC m=+1529.938966169" watchObservedRunningTime="2026-11-03 20:27:06.493182503 +0000 UTC m=+1529.943288550" Nov 03 20:27:07.286958 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:07.286924 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:07.517926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:07.517881 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:07.647883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:07.647788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.103883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:08.103847 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.383549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:08.383464 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.544654 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:08.544609 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.740018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:08.739920 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:08.886690 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:08.886648 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:09.032562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:09.032526 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:09.463275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:09.463173 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:09.649106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:09.649069 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:10.480828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:10.480789 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:11.197577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:11.197540 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:12.128422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:12.128384 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:12.478923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:12.478838 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:27:12.533925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:12.533892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:12.659933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:12.659893 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:13.483475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.476621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:13.838917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.838883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:13.872610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.872581 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p"] Nov 03 20:27:13.876895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.876878 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:13.888155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.888127 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-6e5139\"" Nov 03 20:27:13.889893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.889869 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p"] Nov 03 20:27:13.890097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.890079 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-6e5139-dockercfg-zv2x8\"" Nov 03 20:27:13.914434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.914407 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40815cdd-d7d7-4471-a12c-e16fdccd9617-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p\" (UID: \"40815cdd-d7d7-4471-a12c-e16fdccd9617\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:13.914596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.914552 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbtg\" (UniqueName: \"kubernetes.io/projected/40815cdd-d7d7-4471-a12c-e16fdccd9617-kube-api-access-4bbtg\") pod \"payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p\" (UID: \"40815cdd-d7d7-4471-a12c-e16fdccd9617\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:13.923916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.923887 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp"] Nov 03 20:27:13.928479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.928418 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:13.949353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.949326 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp"] Nov 03 20:27:13.988040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:13.988005 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:14.034471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.023701 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpn8n\" (UniqueName: \"kubernetes.io/projected/b0599233-76dd-4eb9-9b5d-94a815dedd36-kube-api-access-hpn8n\") pod \"payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp\" (UID: \"b0599233-76dd-4eb9-9b5d-94a815dedd36\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:14.034985 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.034953 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbtg\" (UniqueName: \"kubernetes.io/projected/40815cdd-d7d7-4471-a12c-e16fdccd9617-kube-api-access-4bbtg\") pod \"payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p\" (UID: \"40815cdd-d7d7-4471-a12c-e16fdccd9617\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:14.052980 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.052949 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0599233-76dd-4eb9-9b5d-94a815dedd36-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp\" (UID: \"b0599233-76dd-4eb9-9b5d-94a815dedd36\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:14.064693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.064656 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40815cdd-d7d7-4471-a12c-e16fdccd9617-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p\" (UID: \"40815cdd-d7d7-4471-a12c-e16fdccd9617\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:14.086233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.069366 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40815cdd-d7d7-4471-a12c-e16fdccd9617-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p\" (UID: \"40815cdd-d7d7-4471-a12c-e16fdccd9617\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:14.112941 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.112842 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbtg\" (UniqueName: \"kubernetes.io/projected/40815cdd-d7d7-4471-a12c-e16fdccd9617-kube-api-access-4bbtg\") pod \"payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p\" (UID: \"40815cdd-d7d7-4471-a12c-e16fdccd9617\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:14.194485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.194144 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn8n\" (UniqueName: \"kubernetes.io/projected/b0599233-76dd-4eb9-9b5d-94a815dedd36-kube-api-access-hpn8n\") pod \"payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp\" (UID: \"b0599233-76dd-4eb9-9b5d-94a815dedd36\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:14.194485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.194295 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0599233-76dd-4eb9-9b5d-94a815dedd36-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp\" (UID: \"b0599233-76dd-4eb9-9b5d-94a815dedd36\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:14.196498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.196122 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:14.196844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.196820 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0599233-76dd-4eb9-9b5d-94a815dedd36-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp\" (UID: \"b0599233-76dd-4eb9-9b5d-94a815dedd36\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:14.216227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.215763 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpn8n\" (UniqueName: \"kubernetes.io/projected/b0599233-76dd-4eb9-9b5d-94a815dedd36-kube-api-access-hpn8n\") pod \"payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp\" (UID: \"b0599233-76dd-4eb9-9b5d-94a815dedd36\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:14.252987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.251047 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:14.741820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.741792 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p"] Nov 03 20:27:14.860077 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:27:14.860046 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0599233_76dd_4eb9_9b5d_94a815dedd36.slice/crio-e9155046b5f365ab34ca87f70cfa111906d3abf2861f86ebfead038b006ec49a WatchSource:0}: Error finding container e9155046b5f365ab34ca87f70cfa111906d3abf2861f86ebfead038b006ec49a: Status 404 returned error can't find the container with id e9155046b5f365ab34ca87f70cfa111906d3abf2861f86ebfead038b006ec49a Nov 03 20:27:14.862882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:14.862856 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp"] Nov 03 20:27:15.525514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:15.524982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:15.618994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:15.618948 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" event={"ID":"b0599233-76dd-4eb9-9b5d-94a815dedd36","Type":"ContainerStarted","Data":"d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99"} Nov 03 20:27:15.619212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:15.619027 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" event={"ID":"b0599233-76dd-4eb9-9b5d-94a815dedd36","Type":"ContainerStarted","Data":"e9155046b5f365ab34ca87f70cfa111906d3abf2861f86ebfead038b006ec49a"} Nov 03 20:27:15.619212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:15.619062 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:15.621087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:15.621057 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" event={"ID":"40815cdd-d7d7-4471-a12c-e16fdccd9617","Type":"ContainerStarted","Data":"273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449"} Nov 03 20:27:15.621208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:15.621113 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" event={"ID":"40815cdd-d7d7-4471-a12c-e16fdccd9617","Type":"ContainerStarted","Data":"1fef1b76431d844091a5837f7f07b52251f3e15819ba0ab91bbb40e587997d7c"} Nov 03 20:27:15.621272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:15.621257 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:15.641116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:15.641076 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" podStartSLOduration=2.641061957 podStartE2EDuration="2.641061957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:15.638945282 +0000 UTC m=+1539.089051328" watchObservedRunningTime="2026-11-03 20:27:15.641061957 +0000 UTC m=+1539.091168005" Nov 03 20:27:15.658221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:15.658179 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" podStartSLOduration=2.65816572 podStartE2EDuration="2.65816572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:15.656705252 +0000 UTC m=+1539.106811303" watchObservedRunningTime="2026-11-03 20:27:15.65816572 +0000 UTC m=+1539.108271765" Nov 03 20:27:17.442561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:17.442400 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:17.541648 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:27:17.541610 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:27:18.838915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:18.838883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:20.219073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:20.219039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:20.765562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:20.765520 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:21.530383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:21.530345 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:22.207027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:22.206995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:22.621958 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:22.621918 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:23.414432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.414399 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:23.674417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.674332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:23.774385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.774352 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz"] Nov 03 20:27:23.779156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.779131 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.784739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.784713 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-6bd876-openshift-default-dockercfg-r4lds\"" Nov 03 20:27:23.795330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.795306 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz"] Nov 03 20:27:23.815112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.815076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:23.864187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.864145 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-certs\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.864187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.864187 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-credential-socket\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.864493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.864461 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhqh\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-kube-api-access-qmhqh\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.864635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.864604 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-socket\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.864701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.864669 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-envoy\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.864786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.864705 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-token\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.864846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.864812 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-podinfo\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.864893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.864861 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-data\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.864985 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.864906 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istiod-ca-cert\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.971621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.971518 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-podinfo\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.973541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.973515 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-data\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.973758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.973740 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istiod-ca-cert\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.973898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.973883 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-certs\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.974026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.974012 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-credential-socket\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.974329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.974307 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhqh\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-kube-api-access-qmhqh\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.974537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.974518 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-socket\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.974662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.974648 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-envoy\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.974791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.974776 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-token\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:23.975263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.973396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:24.006416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.982999 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-credential-socket\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.006686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.983222 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-data\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.006806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.990638 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istiod-ca-cert\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.006912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.991089 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-socket\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.007003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.991345 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-certs\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.007091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:23.998627 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-envoy\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.014499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.011274 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-token\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.014499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.013524 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhqh\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-kube-api-access-qmhqh\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.015214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.015188 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-podinfo\") pod \"e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.100552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.100521 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:24.139797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.139626 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:24.423908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.423864 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz"] Nov 03 20:27:24.430581 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:27:24.430552 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e00cfda_16b5_4c4f_b5fc_616c2c9ad799.slice/crio-54584f9b083abee6671d817de1312b9a7c16993193148b022c9d1d0425349cd0 WatchSource:0}: Error finding container 54584f9b083abee6671d817de1312b9a7c16993193148b022c9d1d0425349cd0: Status 404 returned error can't find the container with id 54584f9b083abee6671d817de1312b9a7c16993193148b022c9d1d0425349cd0 Nov 03 20:27:24.433912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.433878 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:24.433969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.433952 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:24.434017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.433993 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:24.549691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.549650 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:24.689700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.689594 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" event={"ID":"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799","Type":"ContainerStarted","Data":"4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9"} Nov 03 20:27:24.689700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.689675 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" event={"ID":"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799","Type":"ContainerStarted","Data":"54584f9b083abee6671d817de1312b9a7c16993193148b022c9d1d0425349cd0"} Nov 03 20:27:24.714089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.714038 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" podStartSLOduration=1.714022524 podStartE2EDuration="1.714022524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:24.712475335 +0000 UTC m=+1548.162581382" watchObservedRunningTime="2026-11-03 20:27:24.714022524 +0000 UTC m=+1548.164128570" Nov 03 20:27:24.907272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:24.907209 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:25.101011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:25.100976 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:25.102737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:25.102702 4711 patch_prober.go:29] interesting pod/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 20:27:25.102890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:25.102773 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 20:27:25.815617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:25.815580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:26.105695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:26.105615 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:26.631284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:26.631258 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 20:27:26.631659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:26.631640 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 20:27:26.709492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:26.705739 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:26.709492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:26.706831 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:27:26.898064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:26.897984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:27.083553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:27.083517 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:27.220723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:27.220644 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:27.901970 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:27.901931 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:28.299945 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:28.299907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:28.571536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:28.547497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:29.121728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:29.121660 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:30.079818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.079787 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6"] Nov 03 20:27:30.084376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.084359 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.095708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.095682 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-6e5139-serving-cert\"" Nov 03 20:27:30.098932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.098909 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6"] Nov 03 20:27:30.178843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.178816 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.179248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.178871 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.179248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.179022 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wwj\" (UniqueName: \"kubernetes.io/projected/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-kube-api-access-w4wwj\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.179248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.179056 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.191694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.191668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:30.280377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.280343 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.280591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.280580 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wwj\" (UniqueName: \"kubernetes.io/projected/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-kube-api-access-w4wwj\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.280659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.280618 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.280828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.280809 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.280948 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:27:30.280925 4711 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-serving-cert: secret "maas-api-e2e-shared-b-w3-6e5139-serving-cert" not found Nov 03 20:27:30.281004 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:27:30.280999 4711 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls podName:ecd9ea94-bb2d-44f1-8feb-c68a800f208d nodeName:}" failed. No retries permitted until 2026-11-03 20:27:30.780981053 +0000 UTC m=+1554.231087091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls") pod "maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" (UID: "ecd9ea94-bb2d-44f1-8feb-c68a800f208d") : secret "maas-api-e2e-shared-b-w3-6e5139-serving-cert" not found Nov 03 20:27:30.281543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.281520 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.283393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.283368 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.294912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.294880 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wwj\" (UniqueName: \"kubernetes.io/projected/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-kube-api-access-w4wwj\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.333049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.332947 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:30.663266 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.663183 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:30.787165 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.787125 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:30.794165 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:30.794137 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:31.003769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.003682 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:31.258393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.257424 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:31.332482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.328898 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6"] Nov 03 20:27:31.366476 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:27:31.363978 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd9ea94_bb2d_44f1_8feb_c68a800f208d.slice/crio-53762e7d6dcf60c09b708555a33887110e7a8dd5ca3aa693427917d0baff74c1 WatchSource:0}: Error finding container 53762e7d6dcf60c09b708555a33887110e7a8dd5ca3aa693427917d0baff74c1: Status 404 returned error can't find the container with id 53762e7d6dcf60c09b708555a33887110e7a8dd5ca3aa693427917d0baff74c1 Nov 03 20:27:31.614814 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.614734 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk"] Nov 03 20:27:31.620527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.620505 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.635899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.632486 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3c25bf-openshift-default-dockercfg-bmp7d\"" Nov 03 20:27:31.635899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.632894 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk"] Nov 03 20:27:31.765567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.765526 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.765742 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.765592 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.765794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.765763 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e08425-61fd-49bc-a532-f6b073382774-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.765847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.765811 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fb95\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-kube-api-access-5fb95\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.765895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.765888 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e08425-61fd-49bc-a532-f6b073382774-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.765971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.765959 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.766121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.766101 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.766200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.766182 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.766251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.766204 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.797140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.797106 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" event={"ID":"ecd9ea94-bb2d-44f1-8feb-c68a800f208d","Type":"ContainerStarted","Data":"53762e7d6dcf60c09b708555a33887110e7a8dd5ca3aa693427917d0baff74c1"} Nov 03 20:27:31.867806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.867744 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.867806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.867788 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.867999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.867854 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e08425-61fd-49bc-a532-f6b073382774-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.867999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.867897 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fb95\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-kube-api-access-5fb95\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.867999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.867943 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e08425-61fd-49bc-a532-f6b073382774-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.868153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.868003 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.868153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.868028 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.868153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.868060 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.868153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.868081 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.868671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.868438 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.868806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.868673 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.868877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.868847 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e08425-61fd-49bc-a532-f6b073382774-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.868946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.868883 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.869004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.868969 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.870691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.870661 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.870995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.870973 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e08425-61fd-49bc-a532-f6b073382774-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.880110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.880081 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.880244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.880223 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fb95\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-kube-api-access-5fb95\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:31.943895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:31.943870 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:32.229726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.229696 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk"] Nov 03 20:27:32.230657 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:27:32.230627 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e08425_61fd_49bc_a532_f6b073382774.slice/crio-871de2462ff35f384d132d7274d3c221d4f653868c2aa41585ff5a94d08f18e2 WatchSource:0}: Error finding container 871de2462ff35f384d132d7274d3c221d4f653868c2aa41585ff5a94d08f18e2: Status 404 returned error can't find the container with id 871de2462ff35f384d132d7274d3c221d4f653868c2aa41585ff5a94d08f18e2 Nov 03 20:27:32.233942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.233900 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:32.234048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.234001 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:32.234100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.234043 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:32.400168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.399953 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:32.805480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.805422 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" event={"ID":"ecd9ea94-bb2d-44f1-8feb-c68a800f208d","Type":"ContainerStarted","Data":"b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8"} Nov 03 20:27:32.805690 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.805509 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:32.807064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.807036 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" event={"ID":"40e08425-61fd-49bc-a532-f6b073382774","Type":"ContainerStarted","Data":"34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d"} Nov 03 20:27:32.807177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.807087 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" event={"ID":"40e08425-61fd-49bc-a532-f6b073382774","Type":"ContainerStarted","Data":"871de2462ff35f384d132d7274d3c221d4f653868c2aa41585ff5a94d08f18e2"} Nov 03 20:27:32.829314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.829259 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" podStartSLOduration=2.351001842 podStartE2EDuration="2.829244533s" totalImagesPullingTime="478.242691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:32.825586614 +0000 UTC m=+1556.275692662" watchObservedRunningTime="2026-11-03 20:27:32.829244533 +0000 UTC m=+1556.279350653" Nov 03 20:27:32.853755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.853708 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" podStartSLOduration=1.853694757 podStartE2EDuration="1.853694757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:32.848267255 +0000 UTC m=+1556.298373301" watchObservedRunningTime="2026-11-03 20:27:32.853694757 +0000 UTC m=+1556.303800801" Nov 03 20:27:32.945149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:32.944908 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:33.086589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.086494 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9"] Nov 03 20:27:33.100482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.099717 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.114237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.114191 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9"] Nov 03 20:27:33.149739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.143911 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m"] Nov 03 20:27:33.163291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.162081 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m"] Nov 03 20:27:33.163291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.162174 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.196477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.181057 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bab2c3-openshift-default-dockercfg-bkx62\"" Nov 03 20:27:33.198244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.197809 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3211a0-openshift-default-dockercfg-57hrh\"" Nov 03 20:27:33.209163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.209134 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f223f7-bc76-4228-b969-6c11f753219e-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.209372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.209352 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.209678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.209643 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.209765 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.209807 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.209848 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.209875 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.209898 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.209993 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmr2m\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-kube-api-access-hmr2m\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.210035 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.210088 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzlx\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-kube-api-access-rkzlx\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.210123 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.210147 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.210176 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.210198 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.210219 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f223f7-bc76-4228-b969-6c11f753219e-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.210247 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.210460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.210265 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.311520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.311482 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmr2m\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-kube-api-access-hmr2m\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.311719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.311569 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.311719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.311642 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzlx\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-kube-api-access-rkzlx\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.311719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.311689 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.311911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.311888 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.311998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.311945 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.311998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.311987 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.312112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312022 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f223f7-bc76-4228-b969-6c11f753219e-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.312112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312067 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.312112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312101 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.312264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312189 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f223f7-bc76-4228-b969-6c11f753219e-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.312264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312224 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.312264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312253 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.312420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312283 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.312420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312312 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.312420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312325 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.312420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312407 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.312677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312483 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.312677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312523 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.312829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312671 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.312829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312226 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.312943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312854 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.313004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312943 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.313125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.313098 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.313168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.312317 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.313346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.313324 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f223f7-bc76-4228-b969-6c11f753219e-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.313436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.313376 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.313601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.313580 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.314387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.314361 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.314549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.314412 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.315212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.315187 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f223f7-bc76-4228-b969-6c11f753219e-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.315563 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.315546 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.324080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.324058 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.324531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.324506 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmr2m\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-kube-api-access-hmr2m\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.324617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.324574 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzlx\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-kube-api-access-rkzlx\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.324804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.324786 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.447123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.447049 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:33.514135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.514105 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:33.523731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.523487 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:33.793471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.782647 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9"] Nov 03 20:27:33.798650 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:27:33.798599 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f223f7_bc76_4228_b969_6c11f753219e.slice/crio-9b73ff749c2f77ee65bc889d7bb0d04bde84b36d9510ab0c69d66b0319ad4bcc WatchSource:0}: Error finding container 9b73ff749c2f77ee65bc889d7bb0d04bde84b36d9510ab0c69d66b0319ad4bcc: Status 404 returned error can't find the container with id 9b73ff749c2f77ee65bc889d7bb0d04bde84b36d9510ab0c69d66b0319ad4bcc Nov 03 20:27:33.801737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.801704 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:33.801818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.801783 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:33.801859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.801842 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:33.818377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.818287 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" event={"ID":"78f223f7-bc76-4228-b969-6c11f753219e","Type":"ContainerStarted","Data":"9b73ff749c2f77ee65bc889d7bb0d04bde84b36d9510ab0c69d66b0319ad4bcc"} Nov 03 20:27:33.945846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.945811 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:27:33.945980 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.945870 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" podUID="40e08425-61fd-49bc-a532-f6b073382774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 20:27:33.946162 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:27:33.946135 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ad1145_1b5a_43db_ae2d_704d6e113f61.slice/crio-15a81f72fa860a5ffd9a3999f930017ad51eb4842a02e9825317c213e454dfce WatchSource:0}: Error finding container 15a81f72fa860a5ffd9a3999f930017ad51eb4842a02e9825317c213e454dfce: Status 404 returned error can't find the container with id 15a81f72fa860a5ffd9a3999f930017ad51eb4842a02e9825317c213e454dfce Nov 03 20:27:33.946997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.946972 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m"] Nov 03 20:27:33.949064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.949029 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:33.949130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.949107 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:33.949181 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:33.949154 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:34.728468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:34.728011 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:34.848467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:34.838394 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" event={"ID":"a3ad1145-1b5a-43db-ae2d-704d6e113f61","Type":"ContainerStarted","Data":"7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c"} Nov 03 20:27:34.878651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:34.838492 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" event={"ID":"a3ad1145-1b5a-43db-ae2d-704d6e113f61","Type":"ContainerStarted","Data":"15a81f72fa860a5ffd9a3999f930017ad51eb4842a02e9825317c213e454dfce"} Nov 03 20:27:34.941160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:34.939959 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" event={"ID":"78f223f7-bc76-4228-b969-6c11f753219e","Type":"ContainerStarted","Data":"ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4"} Nov 03 20:27:34.963470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:34.955132 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" podStartSLOduration=1.9551133649999999 podStartE2EDuration="1.955113365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:34.954097369 +0000 UTC m=+1558.404203415" watchObservedRunningTime="2026-11-03 20:27:34.955113365 +0000 UTC m=+1558.405219411" Nov 03 20:27:35.008173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:34.963929 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:27:35.008173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:34.963987 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" podUID="40e08425-61fd-49bc-a532-f6b073382774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 20:27:35.008173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.005737 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" podStartSLOduration=2.005714953 podStartE2EDuration="2.005714953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:34.997658653 +0000 UTC m=+1558.447764701" watchObservedRunningTime="2026-11-03 20:27:35.005714953 +0000 UTC m=+1558.455820997" Nov 03 20:27:35.061507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.056411 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:35.455530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.455007 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:27:35.455530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.455081 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 20:27:35.455769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.455725 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:35.528466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.528244 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:35.528466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.528378 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:27:35.528466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.528423 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 20:27:35.950586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.949568 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:27:35.950586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:35.949630 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" podUID="40e08425-61fd-49bc-a532-f6b073382774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 20:27:36.139038 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:36.139003 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:36.449835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:36.449798 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:27:36.450139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:36.450103 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 20:27:36.581469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:36.560552 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:27:36.581469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:36.560615 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 20:27:36.956472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:36.954169 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:27:36.956472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:36.954228 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" podUID="40e08425-61fd-49bc-a532-f6b073382774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 20:27:37.465540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:37.461841 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:27:37.465540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:37.461893 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 20:27:37.465540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:37.462879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:37.528471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:37.525739 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:27:37.528471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:37.525794 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 20:27:37.828508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:37.816415 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:37.950825 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:37.950795 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 03 20:27:37.951130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:37.951104 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" podUID="40e08425-61fd-49bc-a532-f6b073382774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 03 20:27:37.951595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:37.951044 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:38.449873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:38.448634 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:27:38.449873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:38.448702 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 20:27:38.535713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:38.528043 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:27:38.535713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:38.528113 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 20:27:38.829819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:38.829784 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 20:27:38.842617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:38.841366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:39.049489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:39.048885 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:39.060070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:39.050722 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:39.060070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:39.052561 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:27:39.447639 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:39.447607 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 03 20:27:39.447833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:39.447658 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 03 20:27:39.524433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:39.524398 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:27:39.524868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:39.524477 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 20:27:40.236653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:40.236609 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:40.452806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:40.452750 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:40.453246 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:40.453227 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:40.454214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:40.454191 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:27:40.525298 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:40.525252 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Nov 03 20:27:40.525836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:40.525321 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Nov 03 20:27:41.221748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:41.221708 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:41.529112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:41.529078 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:41.529514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:41.529482 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:41.530388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:41.530368 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:27:42.343586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:42.343494 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:43.016422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:43.016387 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:43.615865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:43.615829 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:44.508148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.508109 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w"] Nov 03 20:27:44.513519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.513493 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.524416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.520847 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-6bd876-serving-cert\"" Nov 03 20:27:44.526524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.526494 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w"] Nov 03 20:27:44.590428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.590391 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:44.612110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.612078 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4chtv\" (UniqueName: \"kubernetes.io/projected/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-kube-api-access-4chtv\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.612110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.612120 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.612332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.612157 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-metrics-certs\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.612332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.612247 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.714789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.714751 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.715092 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.715071 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-metrics-certs\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.715666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.715642 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.716389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.716154 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4chtv\" (UniqueName: \"kubernetes.io/projected/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-kube-api-access-4chtv\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.717000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.716935 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.718826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.718737 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:44.742383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.742336 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-metrics-certs\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.747209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.743103 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.747209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.747155 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4chtv\" (UniqueName: \"kubernetes.io/projected/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-kube-api-access-4chtv\") pod \"maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:44.840912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:44.840825 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:45.161521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:45.159438 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w"] Nov 03 20:27:45.203494 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:27:45.201884 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b8e6f5_b1b7_4ab8_8a54_1f0613e559a9.slice/crio-57d33e30e2f98b77651bb15441988a9264e0fb203f857edf037dbce607618ecd WatchSource:0}: Error finding container 57d33e30e2f98b77651bb15441988a9264e0fb203f857edf037dbce607618ecd: Status 404 returned error can't find the container with id 57d33e30e2f98b77651bb15441988a9264e0fb203f857edf037dbce607618ecd Nov 03 20:27:45.203759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:45.203738 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:27:45.614018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:45.613972 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:46.164694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:46.164648 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" event={"ID":"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9","Type":"ContainerStarted","Data":"099ebb3797e2b1bc5275ef84a99af50bb4631a6c6f36be47fa52b77790e8381c"} Nov 03 20:27:46.164911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:46.164793 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" event={"ID":"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9","Type":"ContainerStarted","Data":"57d33e30e2f98b77651bb15441988a9264e0fb203f857edf037dbce607618ecd"} Nov 03 20:27:46.164997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:46.164952 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:46.190346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:46.190281 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" podStartSLOduration=1.7826461249999999 podStartE2EDuration="2.190265799s" totalImagesPullingTime="407.619674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:46.185562692 +0000 UTC m=+1569.635668736" watchObservedRunningTime="2026-11-03 20:27:46.190265799 +0000 UTC m=+1569.640371845" Nov 03 20:27:46.529630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:46.529525 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:47.217235 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:47.217188 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:47.543737 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:27:47.543698 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:27:47.929745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:47.929590 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:49.001341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:49.001310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:49.922255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:49.922218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:50.824223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:50.824178 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:51.667743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:51.667707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:52.009846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:52.009807 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:52.188688 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:52.188658 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:27:52.708010 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:52.707970 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:53.074016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:53.073971 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:53.441348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:53.441245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:54.315106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:54.315066 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:55.237841 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:55.237793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:56.549561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:56.549521 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:57.209726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:57.209657 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:57.679249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:57.679209 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:57.824278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:57.824242 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:58.590981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:58.590945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:27:59.299501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:27:59.299461 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:00.148700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:00.148664 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:01.303941 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:01.303908 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:02.011264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:02.011232 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:02.539606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:02.539579 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:02.718531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:02.718494 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:02.846666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:02.846575 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:03.503652 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:03.503611 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:04.462879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:04.462843 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:05.216573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:05.216522 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:06.018278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:06.018240 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:06.821891 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:06.821832 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:07.615828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:07.615792 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:08.516240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:08.516155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:09.429439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:09.429397 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:10.203937 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:10.203904 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:10.770916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:10.770880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:10.912364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:10.912329 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:11.693222 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:11.693187 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:12.511192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:12.511157 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:13.404678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:13.404599 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:14.218306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:14.218262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:14.807089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:14.807034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:14.950585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:14.950541 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:15.701282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:15.701246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:16.507473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:16.507406 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:16.920509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:16.920295 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg"] Nov 03 20:28:16.950405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:16.950364 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:16.950857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:16.950824 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg"] Nov 03 20:28:17.079978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.079940 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.080146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.080002 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.080146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.080131 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2kr\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-kube-api-access-jj2kr\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.080225 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.080199 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.080347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.080329 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.080392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.080366 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.080437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.080405 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.080546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.080530 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.080582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.080554 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.181921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.181840 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.181921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.181880 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.181986 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182060 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182082 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182255 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182294 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182325 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182391 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2kr\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-kube-api-access-jj2kr\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182435 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182485 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182611 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182809 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.182910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.182842 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.184384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.184364 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.184589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.184571 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.194478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.194457 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.194679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.194661 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2kr\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-kube-api-access-jj2kr\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.271904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.271871 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:17.303340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.303299 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:17.534153 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:28:17.534121 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:28:17.601620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.601592 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg"] Nov 03 20:28:17.603187 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:28:17.603150 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26ab542_92c0_4424_8dd3_d74dc2839bbb.slice/crio-d303fc8c7e57c76e7421c4a1d9b5086b2ed2536c248bb9e908bcc257857139cb WatchSource:0}: Error finding container d303fc8c7e57c76e7421c4a1d9b5086b2ed2536c248bb9e908bcc257857139cb: Status 404 returned error can't find the container with id d303fc8c7e57c76e7421c4a1d9b5086b2ed2536c248bb9e908bcc257857139cb Nov 03 20:28:17.606138 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.606101 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:17.606224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.606174 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:17.606224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:17.606207 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:18.002221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.002133 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:18.220516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.215963 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s"] Nov 03 20:28:18.236948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.236924 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s"] Nov 03 20:28:18.237332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.237311 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29"] Nov 03 20:28:18.240023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.237245 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.294488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.287712 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29"] Nov 03 20:28:18.294488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.287776 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c"] Nov 03 20:28:18.319467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.313120 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.319765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.319738 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.319942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.319924 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.320101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.320079 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.320225 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.320209 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.320313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.320139 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.320428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.320404 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.320584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.320562 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zm2\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-kube-api-access-r2zm2\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.320792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.320723 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.320878 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.320858 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.327042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.326966 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.334390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.334368 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c"] Nov 03 20:28:18.422376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422342 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.422376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422381 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.422605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422400 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.422605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422427 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.422605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422592 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.422703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422644 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.422703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422683 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.422766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422710 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.422766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422751 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.422829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422764 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.422868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.422853 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zm2\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-kube-api-access-r2zm2\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.423492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423167 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.423492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423258 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.423492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423297 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.423492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423324 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.423492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423328 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.423492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423367 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.423492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423436 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.423492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423485 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.424019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423509 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.424019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423526 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.424019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423589 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.424019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423671 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.424019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423730 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.424019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423773 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27pd\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-kube-api-access-c27pd\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.424019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423803 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.424019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423854 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.424019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.423997 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.424585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.424032 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2lr\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-kube-api-access-qb2lr\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.424585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.424080 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.424585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.424135 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.424821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.424714 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.425006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.424985 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.425854 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.425828 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.435239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.435217 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zm2\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-kube-api-access-r2zm2\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.436051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.436029 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.456516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.456491 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" event={"ID":"d26ab542-92c0-4424-8dd3-d74dc2839bbb","Type":"ContainerStarted","Data":"62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2"} Nov 03 20:28:18.456614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.456540 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" event={"ID":"d26ab542-92c0-4424-8dd3-d74dc2839bbb","Type":"ContainerStarted","Data":"d303fc8c7e57c76e7421c4a1d9b5086b2ed2536c248bb9e908bcc257857139cb"} Nov 03 20:28:18.497020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.496943 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" podStartSLOduration=2.496924896 podStartE2EDuration="2.496924896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:18.493168306 +0000 UTC m=+1601.943274356" watchObservedRunningTime="2026-11-03 20:28:18.496924896 +0000 UTC m=+1601.947030947" Nov 03 20:28:18.525424 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525341 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27pd\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-kube-api-access-c27pd\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.525424 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525388 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.525675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525435 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.525675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525577 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.525675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525614 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2lr\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-kube-api-access-qb2lr\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.525675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525658 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.525906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525714 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.525906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525762 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.525906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525795 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.525906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525839 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.525906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.525868 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.526183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526029 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.526183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526073 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.526183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526103 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.526183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526131 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.526183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526155 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.526674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526218 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.526674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526254 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.526674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526281 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.526674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526363 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.526674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526413 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.526936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526716 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.526936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526908 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.527044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.526959 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.527138 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.527112 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.527215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.527168 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.527279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.527222 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.527328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.527314 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.528361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.528334 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.528559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.528537 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.528754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.528730 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.529159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.529139 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.537839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.537803 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2lr\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-kube-api-access-qb2lr\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.537990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.537970 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27pd\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-kube-api-access-c27pd\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.538505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.538487 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.538687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.538667 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.628659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.628629 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:18.644749 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.644716 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:18.653474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.650202 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:18.955594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.955557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:18.956482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.956434 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s"] Nov 03 20:28:18.957755 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:28:18.957697 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a90ed4c_7a79_4d23_a35f_8e749c9c7f20.slice/crio-7aa51276f1551e6b291d1787fdb5748fe4eb98cae95b7259518ac8c5fe5775e9 WatchSource:0}: Error finding container 7aa51276f1551e6b291d1787fdb5748fe4eb98cae95b7259518ac8c5fe5775e9: Status 404 returned error can't find the container with id 7aa51276f1551e6b291d1787fdb5748fe4eb98cae95b7259518ac8c5fe5775e9 Nov 03 20:28:19.022466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:18.986516 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.023277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.023237 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.023487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.023465 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.052173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.051986 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c"] Nov 03 20:28:19.064319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.064284 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.064417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.064364 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.064417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.064397 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.112489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.112423 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29"] Nov 03 20:28:19.118987 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:28:19.118953 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b6a1c4_e8bc_44c8_8a59_bc02e35397cf.slice/crio-6da250c15986c52e202e89fe33d0655269f6aefd240988eb7cbe4a7e4ae9de25 WatchSource:0}: Error finding container 6da250c15986c52e202e89fe33d0655269f6aefd240988eb7cbe4a7e4ae9de25: Status 404 returned error can't find the container with id 6da250c15986c52e202e89fe33d0655269f6aefd240988eb7cbe4a7e4ae9de25 Nov 03 20:28:19.124657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.123883 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.124657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.123973 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.124657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.124019 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:19.274007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.273965 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:19.277870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.277846 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:28:19.278054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.278031 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 20:28:19.482475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.481161 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" event={"ID":"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20","Type":"ContainerStarted","Data":"373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e"} Nov 03 20:28:19.482475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.481233 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" event={"ID":"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20","Type":"ContainerStarted","Data":"7aa51276f1551e6b291d1787fdb5748fe4eb98cae95b7259518ac8c5fe5775e9"} Nov 03 20:28:19.500732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.500693 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" event={"ID":"9a228a8b-66b7-4c86-ac95-c24bb5f3043c","Type":"ContainerStarted","Data":"219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067"} Nov 03 20:28:19.501030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.500993 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" event={"ID":"9a228a8b-66b7-4c86-ac95-c24bb5f3043c","Type":"ContainerStarted","Data":"339229833f3716548caf61af2b9d14d161392498e87c6090cab94ce07c71c599"} Nov 03 20:28:19.516867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.516832 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" event={"ID":"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf","Type":"ContainerStarted","Data":"592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea"} Nov 03 20:28:19.517085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.517066 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" event={"ID":"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf","Type":"ContainerStarted","Data":"6da250c15986c52e202e89fe33d0655269f6aefd240988eb7cbe4a7e4ae9de25"} Nov 03 20:28:19.533780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.532341 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" podStartSLOduration=1.532323468 podStartE2EDuration="1.532323468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:19.519780793 +0000 UTC m=+1602.969886840" watchObservedRunningTime="2026-11-03 20:28:19.532323468 +0000 UTC m=+1602.982429514" Nov 03 20:28:19.561707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.561660 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podStartSLOduration=1.561641219 podStartE2EDuration="1.561641219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:19.543527972 +0000 UTC m=+1602.993634022" watchObservedRunningTime="2026-11-03 20:28:19.561641219 +0000 UTC m=+1603.011747266" Nov 03 20:28:19.633150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.633063 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:19.634749 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.634255 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" podStartSLOduration=1.6342349600000001 podStartE2EDuration="1.63423496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:19.583507369 +0000 UTC m=+1603.033613414" watchObservedRunningTime="2026-11-03 20:28:19.63423496 +0000 UTC m=+1603.084341008" Nov 03 20:28:19.645600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.645573 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:19.655805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:19.655778 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:20.019568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:20.013261 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:20.288600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:20.278793 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:28:20.288600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:20.278871 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 20:28:20.747724 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:20.747633 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:28:20.747975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:20.747946 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 20:28:20.786226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:20.786189 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:28:20.791930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:20.786457 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 20:28:20.843855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:20.843814 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:28:20.847468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:20.844077 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 20:28:21.156597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.156552 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:21.312477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.301428 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:28:21.312477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.301519 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 20:28:21.391248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.391205 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:21.428528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.426486 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:28:21.428528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.427024 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:21.524479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.515496 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:21.558475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.525655 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:21.597501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.597437 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:21.607775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.606205 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:21.746565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.746468 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:28:21.840679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.840621 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 20:28:21.841484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.836544 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:21.842066 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.750286 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:28:21.842266 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.842228 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 20:28:21.842530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.752348 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:28:21.842686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.842666 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 20:28:21.951471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:21.950593 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:22.257471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.256652 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:22.282471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.276321 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:28:22.282471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.276377 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 20:28:22.419472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.419328 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:28:22.454472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.419998 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:28:22.494472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.456684 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:22.525672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.461563 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:22.526568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.526542 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:28:22.526759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.526745 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:28:22.591471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.584530 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:22.591471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.584975 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:22.591471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.585298 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:28:22.591471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.585638 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:28:22.591471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.585694 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:28:22.630471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.629862 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:22.630471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.630257 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:28:22.635477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.631461 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:28:22.635477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.631510 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 20:28:22.649901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.649857 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:28:22.650155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.650127 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 20:28:22.664981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.664950 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:28:22.666482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.666435 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 20:28:22.847017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.846921 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:28:22.911489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:22.907720 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:23.041467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.034411 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:23.276531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.257682 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:23.276531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.274289 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:28:23.276531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.274345 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:28:23.298974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.298933 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:28:23.327865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.327823 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 20:28:23.430475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.429544 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:23.546638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.546539 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:23.552466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.548470 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:28:23.659643 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.641548 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:23.679691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.665889 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:28:23.679691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.666211 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:28:23.680868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.680834 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:28:23.703085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.703047 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 20:28:23.724286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.709408 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:28:23.724286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.709482 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 20:28:23.927496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.927376 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:28:23.927784 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.927739 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 20:28:24.021471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.949595 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:24.021471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:23.950784 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:28:24.373836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.373799 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:28:24.469581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.469536 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:28:24.989251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.530079 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:28:25.052616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.530308 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded" Nov 03 20:28:25.053149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.556000 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.053423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.663601 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.053668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.685759 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.053871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.719196 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:28:25.084293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.794764 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded" Nov 03 20:28:25.084548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.828922 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.084801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.084733 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 03 20:28:25.084992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:24.954586 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.086516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.002753 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.086705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.086678 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 20:28:25.251651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.163940 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.251894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.251864 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 20:28:25.281294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.281258 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:25.692424 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.690566 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.692424 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.690632 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 20:28:25.749584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.739130 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.749584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.739194 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 20:28:25.828704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.828646 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:25.835481 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:28:25.832823 4711 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.085s" Nov 03 20:28:25.835481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.832940 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:25.853484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.836920 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:26.038529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.986354 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:28:26.064431 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.064391 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.064959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.986418 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:28:26.065177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.985552 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.069466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:25.989279 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.088626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.057605 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.107178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.092186 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.126468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.057654 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Nov 03 20:28:26.127145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.057572 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.182470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.116632 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:28:26.212470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.211680 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 20:28:26.212470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.121374 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.212470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.057635 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:28:26.212470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.121411 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 20:28:26.426672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.426575 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:26.599192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.599113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:26.653998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.653963 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:28:26.657109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.657074 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 20:28:26.657713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.654230 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Nov 03 20:28:26.658135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.658114 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Nov 03 20:28:26.658287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.654272 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:28:26.658417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.658399 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 03 20:28:26.954474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.953765 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 20:28:26.970063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:26.959097 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 20:28:27.011914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.010613 4711 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.085588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.085540 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 20:28:27.139151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.132633 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:28:27.140232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.140210 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:28:27.140939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.138353 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.141306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.138389 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.182634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.182586 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:27.183756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.183731 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 20:28:27.197721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.197282 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.224535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.216924 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" podUID="d59d95c7-b4a1-4307-8227-09e1a43822cb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.271497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.258619 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:28:27.526108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.526076 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 20:28:27.527811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.527781 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 20:28:27.528428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.528411 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5" Nov 03 20:28:27.529212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.529181 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 20:28:27.530724 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.530699 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 20:28:27.530888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.530876 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 20:28:27.635429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.635401 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:27.636232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.636172 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:27.638480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.638313 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:28:27.654435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.654402 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 03 20:28:27.654615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.654501 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 03 20:28:27.792945 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.792866 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:27.795106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.795085 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:27.798531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:27.798512 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:28:28.610243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:28.610157 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:28.668284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:28.668242 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:28.670794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:28.670230 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:28.671234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:28.671215 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:28:30.110660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:30.110606 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:31.099662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:31.099625 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:31.554949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:31.554913 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:31.923682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:31.923605 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm"] Nov 03 20:28:31.928752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:31.928723 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:31.947284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:31.947256 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm"] Nov 03 20:28:32.098382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.098350 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzt55\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-kube-api-access-hzt55\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.098382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.098394 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.098676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.098526 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.098728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.098712 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.098777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.098765 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.098850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.098827 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.098886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.098864 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.098920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.098904 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.099003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.098990 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.200953 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201048 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzt55\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-kube-api-access-hzt55\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201066 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201108 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201244 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201290 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201315 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201345 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201375 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201403 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201703 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-credential-socket\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201756 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-certs\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.201844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.201789 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-data\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.202144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.202122 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istiod-ca-cert\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.203653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.203608 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-envoy\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.203801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.203754 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-podinfo\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.215699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.215589 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-token\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.215828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.215798 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzt55\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-kube-api-access-hzt55\") pod \"e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.223026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.222980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:32.254153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.254122 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:32.547614 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:28:32.547585 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b58e1c_d802_4a1d_9833_c2557d8128e4.slice/crio-1af5dfb7d9c948fc88bfba6228a82bf78ea73a61f378ec3c1d6d1ef74cd36d52 WatchSource:0}: Error finding container 1af5dfb7d9c948fc88bfba6228a82bf78ea73a61f378ec3c1d6d1ef74cd36d52: Status 404 returned error can't find the container with id 1af5dfb7d9c948fc88bfba6228a82bf78ea73a61f378ec3c1d6d1ef74cd36d52 Nov 03 20:28:32.547890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.547867 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm"] Nov 03 20:28:32.551086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.551015 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:32.551187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.551126 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:32.551187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.551158 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:32.942730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.942619 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:32.963335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.960273 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" event={"ID":"a0b58e1c-d802-4a1d-9833-c2557d8128e4","Type":"ContainerStarted","Data":"65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf"} Nov 03 20:28:32.963335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.960339 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" event={"ID":"a0b58e1c-d802-4a1d-9833-c2557d8128e4","Type":"ContainerStarted","Data":"1af5dfb7d9c948fc88bfba6228a82bf78ea73a61f378ec3c1d6d1ef74cd36d52"} Nov 03 20:28:32.992690 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:32.992630 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" podStartSLOduration=1.9926135299999999 podStartE2EDuration="1.99261353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:32.988106154 +0000 UTC m=+1616.438212201" watchObservedRunningTime="2026-11-03 20:28:32.99261353 +0000 UTC m=+1616.442719575" Nov 03 20:28:33.208517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.208148 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv"] Nov 03 20:28:33.225470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.216739 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.228289 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.228391 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.228432 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf17d6-5203-4647-927f-b5abc73ad8e9-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.228491 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.228775 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.228889 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.229032 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.229165 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqtm5\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-kube-api-access-rqtm5\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.229194 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.257986 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.258121 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:33.264485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.258173 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 20:28:33.311471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.306525 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv"] Nov 03 20:28:33.339337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.337350 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqtm5\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-kube-api-access-rqtm5\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.339337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.337408 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.339337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.337486 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.339337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.337573 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.339337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.337631 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf17d6-5203-4647-927f-b5abc73ad8e9-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.339337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.337682 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.339337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.337835 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.339337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.337952 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.339337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.338053 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.353667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.340304 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-certs\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.353667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.341358 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf17d6-5203-4647-927f-b5abc73ad8e9-istiod-ca-cert\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.361907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.361858 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-credential-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.394047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.394009 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-data\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.394967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.394930 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-socket\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.455066 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.452715 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-envoy\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.515167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.493577 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-podinfo\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.515167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.494130 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqtm5\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-kube-api-access-rqtm5\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.515167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.494468 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-token\") pod \"e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.641764 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.641726 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:33.740469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:33.732753 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:34.051564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:34.049985 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv"] Nov 03 20:28:34.051564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:34.051195 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:34.051564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:34.051304 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:34.051564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:34.051358 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:34.255212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:34.255181 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:34.255397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:34.255230 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 20:28:34.501229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:34.501139 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:35.048331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:35.048293 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" event={"ID":"08bf17d6-5203-4647-927f-b5abc73ad8e9","Type":"ContainerStarted","Data":"31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa"} Nov 03 20:28:35.050080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:35.050050 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" event={"ID":"08bf17d6-5203-4647-927f-b5abc73ad8e9","Type":"ContainerStarted","Data":"67ade914644eb6993ea70e5a0160e137d05d5c8a023583d1f237be27d305f9ee"} Nov 03 20:28:35.090494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:35.078610 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" podStartSLOduration=2.07858956 podStartE2EDuration="2.07858956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:35.075920086 +0000 UTC m=+1618.526026144" watchObservedRunningTime="2026-11-03 20:28:35.07858956 +0000 UTC m=+1618.528695608" Nov 03 20:28:35.254693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:35.254566 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:35.254693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:35.254619 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 20:28:35.386531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:35.386141 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:35.642231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:35.642099 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:35.644055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:35.644012 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 20:28:35.644217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:35.644085 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 20:28:36.255193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:36.255159 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:36.255633 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:36.255203 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 20:28:36.395467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:36.390410 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:36.642250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:36.642169 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 20:28:36.642250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:36.642220 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 20:28:37.137068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:37.137033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:37.260513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:37.259820 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:37.260513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:37.259877 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 20:28:37.642414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:37.642376 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 20:28:37.642617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:37.642455 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 20:28:38.004537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:38.003342 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:38.255343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:38.255262 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:38.255343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:38.255318 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 20:28:38.642498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:38.642397 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 20:28:38.642498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:38.642461 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 20:28:38.745716 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:38.745678 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:39.255004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:39.254973 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 03 20:28:39.255187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:39.255026 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 03 20:28:39.642930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:39.642837 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 20:28:39.642930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:39.642893 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 20:28:39.862328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:39.862291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:40.314471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:40.310754 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:40.314471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:40.312518 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:40.332476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:40.331861 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:28:40.709529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:40.708418 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 20:28:40.709529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:40.708498 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 20:28:41.562436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:41.562397 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:42.205750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:42.205717 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:42.207839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:42.207818 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:42.208559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:42.208540 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:28:42.627306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:42.620304 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:43.417539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:43.417496 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:44.221906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:44.221851 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:45.023119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:45.023080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:45.910327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:45.906686 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:46.823863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:46.823768 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:47.535354 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:28:47.535317 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:28:47.616143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:47.616103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:48.320422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:48.320392 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:48.810532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:48.810494 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:49.067844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:49.067752 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:49.826483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:49.824538 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:50.493898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:50.493859 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:51.287767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:51.287728 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:52.020107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:52.020072 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:52.420143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:52.420112 4711 generic.go:356] "Generic (PLEG): container finished" podID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerID="4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9" exitCode=0 Nov 03 20:28:52.420526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:52.420149 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" event={"ID":"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799","Type":"ContainerDied","Data":"4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9"} Nov 03 20:28:52.420590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:52.420575 4711 scope.go:122] "RemoveContainer" containerID="4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9" Nov 03 20:28:52.422318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:52.422289 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:52.422376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:52.422359 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:52.422412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:52.422391 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:52.722267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:52.722172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:53.429020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:53.428986 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" event={"ID":"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799","Type":"ContainerStarted","Data":"66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b"} Nov 03 20:28:53.493984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:53.493949 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:54.102003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:54.101956 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:28:54.103283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:54.103250 4711 patch_prober.go:29] interesting pod/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 20:28:54.103464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:54.103326 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 20:28:54.296071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:54.296023 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:55.013536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:55.013498 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:55.105515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:55.105480 4711 patch_prober.go:29] interesting pod/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 20:28:55.105778 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:55.105751 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 20:28:55.764956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:55.764916 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:56.101003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:56.100910 4711 patch_prober.go:29] interesting pod/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 20:28:56.101003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:56.100966 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 20:28:56.627403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:56.627362 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:56.741473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:56.733589 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:28:57.101025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:57.100994 4711 patch_prober.go:29] interesting pod/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 20:28:57.101528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:57.101067 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 20:28:57.262748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:57.262716 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:57.403531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:57.402915 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:58.100408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:58.100369 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:58.101978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:58.101946 4711 patch_prober.go:29] interesting pod/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 20:28:58.102389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:58.102011 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 20:28:58.852348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:58.852304 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:28:59.106999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:59.106923 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:28:59.108127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:59.108101 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:28:59.522204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:28:59.522171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:00.339615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:00.339576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:01.228256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:01.228215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:02.223865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:02.223804 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:03.326478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:03.326429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:04.015144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:04.015090 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:04.713572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:04.713536 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:05.526366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:05.526323 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:06.235543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:06.235504 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:06.920280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:06.920192 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:07.606744 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:07.606670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:08.318602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:08.318568 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:09.122799 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:09.122764 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:09.903656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:09.903618 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:10.681527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:10.681490 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:11.373296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:11.373261 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:12.193946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:12.193888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:12.446327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:12.446244 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:12.576773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:12.576735 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:12.915796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:12.915759 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:13.695364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:13.695336 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:14.395832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:14.395789 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:15.150935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:15.150898 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:15.899943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:15.899903 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:16.698191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:16.698158 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:17.535543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:17.533109 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:17.596938 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:29:17.596813 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:29:18.194839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:18.194793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:18.896866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:18.896820 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:19.724660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:19.724627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:20.583485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:20.583439 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:21.330440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:21.323319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:22.195108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:22.195059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:22.945455 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:22.945408 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:23.732760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:23.732725 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:24.525427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:24.525392 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:25.318551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:25.318426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:26.052162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:26.052126 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:26.826316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:26.826281 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:27.512858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:27.512823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:28.294070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:28.294026 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:28.995369 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:28.995334 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:29.716347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:29.716281 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:30.513804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:30.513764 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:31.206675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:31.206633 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:31.892656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:31.892613 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:32.717020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:32.716982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:33.418135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:33.418099 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:34.119693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:34.119657 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:34.767197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:34.767157 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:35.429004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:35.428919 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:36.158606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:36.158572 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:37.006881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:37.006841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:38.004606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:38.004567 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:38.828946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:38.828910 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:39.599827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:39.599790 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:40.434468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:40.432155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:41.094409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:41.094369 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:41.749216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:41.749185 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:42.420692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:42.420656 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:43.183052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:43.183007 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:43.903230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:43.903190 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:44.596997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:44.596958 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:45.278857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:45.278807 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:45.960109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:45.960055 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:46.626356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:46.626321 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:47.540183 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:29:47.540144 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:29:47.621055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:47.621021 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:48.289187 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:48.289149 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:49.125179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:49.125010 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:49.854420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:49.854373 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:50.631104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:50.631059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:51.311349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:51.311313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:52.003667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:52.003630 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:52.689366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:52.689327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:53.393860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:53.393825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:54.124883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:54.124838 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:54.818641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:54.818604 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:55.832961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:55.832926 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:56.704477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:56.703415 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:57.310599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:57.310567 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:57.509834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:57.509801 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:58.253483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:58.253428 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:59.015093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:59.015053 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:29:59.766403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:29:59.766364 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:00.201131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.201036 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j"] Nov 03 20:30:00.207058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.207032 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j"] Nov 03 20:30:00.207177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.207142 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:00.211979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.211954 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:00.234898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.234854 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j"] Nov 03 20:30:00.243367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.241159 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j"] Nov 03 20:30:00.251081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.251052 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs"] Nov 03 20:30:00.267739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.267691 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:00.271412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.271386 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k"] Nov 03 20:30:00.277349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.277323 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:00.279975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.279948 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs"] Nov 03 20:30:00.283920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.283893 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k"] Nov 03 20:30:00.287872 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.287840 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:30:00.315365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.315331 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mdf\" (UniqueName: \"kubernetes.io/projected/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-kube-api-access-27mdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j\" (UID: \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:00.315557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.315400 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j\" (UID: \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:00.315793 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.315775 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrpj\" (UniqueName: \"kubernetes.io/projected/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-kube-api-access-dkrpj\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j\" (UID: \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:00.315869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.315837 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j\" (UID: \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:00.417250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.417212 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bts\" (UniqueName: \"kubernetes.io/projected/4b295009-4e05-4150-b515-9c04a0040631-kube-api-access-w4bts\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs\" (UID: \"4b295009-4e05-4150-b515-9c04a0040631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:00.417495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.417258 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j\" (UID: \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:00.417599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.417528 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66805357-9d48-4395-b3ab-ae68309d7830-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k\" (UID: \"66805357-9d48-4395-b3ab-ae68309d7830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:00.417677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.417655 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlgd2\" (UniqueName: \"kubernetes.io/projected/66805357-9d48-4395-b3ab-ae68309d7830-kube-api-access-tlgd2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k\" (UID: \"66805357-9d48-4395-b3ab-ae68309d7830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:00.417827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.417804 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b295009-4e05-4150-b515-9c04a0040631-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs\" (UID: \"4b295009-4e05-4150-b515-9c04a0040631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:00.417882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.417865 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrpj\" (UniqueName: \"kubernetes.io/projected/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-kube-api-access-dkrpj\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j\" (UID: \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:00.417948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.417931 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j\" (UID: \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:00.418008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.417988 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j\" (UID: \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:00.418089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.418070 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mdf\" (UniqueName: \"kubernetes.io/projected/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-kube-api-access-27mdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j\" (UID: \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:00.418575 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.418551 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j\" (UID: \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:00.430687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.430665 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mdf\" (UniqueName: \"kubernetes.io/projected/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-kube-api-access-27mdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j\" (UID: \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:00.430860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.430841 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrpj\" (UniqueName: \"kubernetes.io/projected/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-kube-api-access-dkrpj\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j\" (UID: \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:00.519855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.519819 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bts\" (UniqueName: \"kubernetes.io/projected/4b295009-4e05-4150-b515-9c04a0040631-kube-api-access-w4bts\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs\" (UID: \"4b295009-4e05-4150-b515-9c04a0040631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:00.520035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.519955 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66805357-9d48-4395-b3ab-ae68309d7830-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k\" (UID: \"66805357-9d48-4395-b3ab-ae68309d7830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:00.520035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.520011 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlgd2\" (UniqueName: \"kubernetes.io/projected/66805357-9d48-4395-b3ab-ae68309d7830-kube-api-access-tlgd2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k\" (UID: \"66805357-9d48-4395-b3ab-ae68309d7830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:00.520110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.520080 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b295009-4e05-4150-b515-9c04a0040631-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs\" (UID: \"4b295009-4e05-4150-b515-9c04a0040631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:00.520693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.520673 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66805357-9d48-4395-b3ab-ae68309d7830-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k\" (UID: \"66805357-9d48-4395-b3ab-ae68309d7830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:00.520815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.520796 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b295009-4e05-4150-b515-9c04a0040631-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs\" (UID: \"4b295009-4e05-4150-b515-9c04a0040631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:00.531905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.531878 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlgd2\" (UniqueName: \"kubernetes.io/projected/66805357-9d48-4395-b3ab-ae68309d7830-kube-api-access-tlgd2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k\" (UID: \"66805357-9d48-4395-b3ab-ae68309d7830\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:00.536242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.536218 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bts\" (UniqueName: \"kubernetes.io/projected/4b295009-4e05-4150-b515-9c04a0040631-kube-api-access-w4bts\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs\" (UID: \"4b295009-4e05-4150-b515-9c04a0040631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:00.540140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.540118 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:00.560857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.560832 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:00.596099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.596008 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:00.599954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.599922 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:00.611718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.611635 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:00.848425 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:30:00.848381 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f0e4f8_e699_4d18_8cab_7ebad68c4ee8.slice/crio-bc7403d4e7574c36220a7f2d484d9a882398ef4becc07787c73e5ee72e556410 WatchSource:0}: Error finding container bc7403d4e7574c36220a7f2d484d9a882398ef4becc07787c73e5ee72e556410: Status 404 returned error can't find the container with id bc7403d4e7574c36220a7f2d484d9a882398ef4becc07787c73e5ee72e556410 Nov 03 20:30:00.849209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.849186 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j"] Nov 03 20:30:00.965487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:00.965457 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j"] Nov 03 20:30:00.965624 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:30:00.965604 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6764eac5_a3cb_4b6c_8fd3_0e2c7d35f665.slice/crio-041cf0e3ba41de738a2e964032972075ac1d9ea5b31d7dd04a6a55743595f023 WatchSource:0}: Error finding container 041cf0e3ba41de738a2e964032972075ac1d9ea5b31d7dd04a6a55743595f023: Status 404 returned error can't find the container with id 041cf0e3ba41de738a2e964032972075ac1d9ea5b31d7dd04a6a55743595f023 Nov 03 20:30:01.014624 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:30:01.014598 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b295009_4e05_4150_b515_9c04a0040631.slice/crio-589cfdbbaab719e1714738696d1cd42d84c7edf112252174ad55ee928b810d90 WatchSource:0}: Error finding container 589cfdbbaab719e1714738696d1cd42d84c7edf112252174ad55ee928b810d90: Status 404 returned error can't find the container with id 589cfdbbaab719e1714738696d1cd42d84c7edf112252174ad55ee928b810d90 Nov 03 20:30:01.015360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:01.015337 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs"] Nov 03 20:30:01.050024 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:30:01.049994 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66805357_9d48_4395_b3ab_ae68309d7830.slice/crio-2151006f227bb3871064392845c44f77fe076f77c6395c330571cb7a8fa1b6b7 WatchSource:0}: Error finding container 2151006f227bb3871064392845c44f77fe076f77c6395c330571cb7a8fa1b6b7: Status 404 returned error can't find the container with id 2151006f227bb3871064392845c44f77fe076f77c6395c330571cb7a8fa1b6b7 Nov 03 20:30:01.050735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:01.050711 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k"] Nov 03 20:30:01.160731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:01.160689 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" event={"ID":"66805357-9d48-4395-b3ab-ae68309d7830","Type":"ContainerStarted","Data":"2151006f227bb3871064392845c44f77fe076f77c6395c330571cb7a8fa1b6b7"} Nov 03 20:30:01.162107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:01.162080 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" event={"ID":"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665","Type":"ContainerStarted","Data":"041cf0e3ba41de738a2e964032972075ac1d9ea5b31d7dd04a6a55743595f023"} Nov 03 20:30:01.163438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:01.163414 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" event={"ID":"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8","Type":"ContainerStarted","Data":"bc7403d4e7574c36220a7f2d484d9a882398ef4becc07787c73e5ee72e556410"} Nov 03 20:30:01.164703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:01.164685 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" event={"ID":"4b295009-4e05-4150-b515-9c04a0040631","Type":"ContainerStarted","Data":"589cfdbbaab719e1714738696d1cd42d84c7edf112252174ad55ee928b810d90"} Nov 03 20:30:01.311348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:01.311312 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:02.169402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.168811 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:02.221585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.219411 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" event={"ID":"66805357-9d48-4395-b3ab-ae68309d7830","Type":"ContainerStarted","Data":"36c4d4bf87b77b95d15bd9c0287997b0e5bd0d0c893b9f58e90abafa189b3351"} Nov 03 20:30:02.222305 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.222211 4711 generic.go:356] "Generic (PLEG): container finished" podID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerID="172190ed008f063067817f3682e0fad0fb7053f3c6fa6104dd456b915f0b63ff" exitCode=60 Nov 03 20:30:02.222305 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.222244 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" event={"ID":"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665","Type":"ContainerDied","Data":"172190ed008f063067817f3682e0fad0fb7053f3c6fa6104dd456b915f0b63ff"} Nov 03 20:30:02.225564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.225544 4711 scope.go:122] "RemoveContainer" containerID="172190ed008f063067817f3682e0fad0fb7053f3c6fa6104dd456b915f0b63ff" Nov 03 20:30:02.228025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.227718 4711 generic.go:356] "Generic (PLEG): container finished" podID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerID="6fdde45643dfde1d0b4fdf77e817b46502ed604bb547ab652c912a28a5518e28" exitCode=60 Nov 03 20:30:02.228025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.227901 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" event={"ID":"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8","Type":"ContainerDied","Data":"6fdde45643dfde1d0b4fdf77e817b46502ed604bb547ab652c912a28a5518e28"} Nov 03 20:30:02.228719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.228416 4711 scope.go:122] "RemoveContainer" containerID="6fdde45643dfde1d0b4fdf77e817b46502ed604bb547ab652c912a28a5518e28" Nov 03 20:30:02.238344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.236312 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" event={"ID":"4b295009-4e05-4150-b515-9c04a0040631","Type":"ContainerStarted","Data":"2017efa9c1493a0eb4cbfc69383934291ae3e15d0b983ea97bdf8c16d3a6f623"} Nov 03 20:30:02.238344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.236617 4711 scope.go:122] "RemoveContainer" containerID="2017efa9c1493a0eb4cbfc69383934291ae3e15d0b983ea97bdf8c16d3a6f623" Nov 03 20:30:02.245022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:02.244973 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" podStartSLOduration=2.244956121 podStartE2EDuration="2.244956121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.241849264 +0000 UTC m=+1705.691955312" watchObservedRunningTime="2026-11-03 20:30:02.244956121 +0000 UTC m=+1705.695062169" Nov 03 20:30:03.100314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.100226 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:03.251179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.251137 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" event={"ID":"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8","Type":"ContainerStarted","Data":"2fe82b01fad366b272884022a3af6b376a531b125295768cf2714ec27f1ecfd5"} Nov 03 20:30:03.253184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.253158 4711 generic.go:356] "Generic (PLEG): container finished" podID="4b295009-4e05-4150-b515-9c04a0040631" containerID="2017efa9c1493a0eb4cbfc69383934291ae3e15d0b983ea97bdf8c16d3a6f623" exitCode=60 Nov 03 20:30:03.253327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.253232 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" event={"ID":"4b295009-4e05-4150-b515-9c04a0040631","Type":"ContainerDied","Data":"2017efa9c1493a0eb4cbfc69383934291ae3e15d0b983ea97bdf8c16d3a6f623"} Nov 03 20:30:03.253327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.253279 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" event={"ID":"4b295009-4e05-4150-b515-9c04a0040631","Type":"ContainerStarted","Data":"292f3cca562161110f3c233a32290b7ed5727d9399df2e183654560435a1a921"} Nov 03 20:30:03.255017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.254994 4711 generic.go:356] "Generic (PLEG): container finished" podID="66805357-9d48-4395-b3ab-ae68309d7830" containerID="36c4d4bf87b77b95d15bd9c0287997b0e5bd0d0c893b9f58e90abafa189b3351" exitCode=60 Nov 03 20:30:03.255154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.255030 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" event={"ID":"66805357-9d48-4395-b3ab-ae68309d7830","Type":"ContainerDied","Data":"36c4d4bf87b77b95d15bd9c0287997b0e5bd0d0c893b9f58e90abafa189b3351"} Nov 03 20:30:03.255382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.255361 4711 scope.go:122] "RemoveContainer" containerID="36c4d4bf87b77b95d15bd9c0287997b0e5bd0d0c893b9f58e90abafa189b3351" Nov 03 20:30:03.257192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.257173 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" event={"ID":"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665","Type":"ContainerStarted","Data":"a85ea05ecb972081dcdcced656f694400499c72c38afdb2305be4adb0e72f95c"} Nov 03 20:30:03.271637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.271590 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" podStartSLOduration=3.271571275 podStartE2EDuration="3.271571275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.270837037 +0000 UTC m=+1706.720943086" watchObservedRunningTime="2026-11-03 20:30:03.271571275 +0000 UTC m=+1706.721677320" Nov 03 20:30:03.296954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.296904 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" podStartSLOduration=3.2968909650000002 podStartE2EDuration="3.296890965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.293487628 +0000 UTC m=+1706.743593693" watchObservedRunningTime="2026-11-03 20:30:03.296890965 +0000 UTC m=+1706.746997080" Nov 03 20:30:03.333226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.333159 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" podStartSLOduration=3.333145362 podStartE2EDuration="3.333145362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:03.328916672 +0000 UTC m=+1706.779022730" watchObservedRunningTime="2026-11-03 20:30:03.333145362 +0000 UTC m=+1706.783251451" Nov 03 20:30:03.913122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:03.913083 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:04.267384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.267356 4711 generic.go:356] "Generic (PLEG): container finished" podID="66805357-9d48-4395-b3ab-ae68309d7830" containerID="ada189c43c37fb5bcec9d11fd4294ec1af80dc9a251a31ebbb62509f18cf18f1" exitCode=60 Nov 03 20:30:04.267829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.267464 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" event={"ID":"66805357-9d48-4395-b3ab-ae68309d7830","Type":"ContainerDied","Data":"ada189c43c37fb5bcec9d11fd4294ec1af80dc9a251a31ebbb62509f18cf18f1"} Nov 03 20:30:04.267829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.267558 4711 scope.go:122] "RemoveContainer" containerID="36c4d4bf87b77b95d15bd9c0287997b0e5bd0d0c893b9f58e90abafa189b3351" Nov 03 20:30:04.267980 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.267962 4711 scope.go:122] "RemoveContainer" containerID="ada189c43c37fb5bcec9d11fd4294ec1af80dc9a251a31ebbb62509f18cf18f1" Nov 03 20:30:04.268480 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:04.268412 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k_odh-ai-gateway-infra(66805357-9d48-4395-b3ab-ae68309d7830)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" podUID="66805357-9d48-4395-b3ab-ae68309d7830" Nov 03 20:30:04.269649 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.269625 4711 generic.go:356] "Generic (PLEG): container finished" podID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerID="a85ea05ecb972081dcdcced656f694400499c72c38afdb2305be4adb0e72f95c" exitCode=60 Nov 03 20:30:04.269735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.269701 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" event={"ID":"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665","Type":"ContainerDied","Data":"a85ea05ecb972081dcdcced656f694400499c72c38afdb2305be4adb0e72f95c"} Nov 03 20:30:04.270014 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.269987 4711 scope.go:122] "RemoveContainer" containerID="a85ea05ecb972081dcdcced656f694400499c72c38afdb2305be4adb0e72f95c" Nov 03 20:30:04.270383 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:04.270360 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j_odh-ai-gateway-infra(6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" Nov 03 20:30:04.271484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.271461 4711 generic.go:356] "Generic (PLEG): container finished" podID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerID="2fe82b01fad366b272884022a3af6b376a531b125295768cf2714ec27f1ecfd5" exitCode=60 Nov 03 20:30:04.271581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.271530 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" event={"ID":"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8","Type":"ContainerDied","Data":"2fe82b01fad366b272884022a3af6b376a531b125295768cf2714ec27f1ecfd5"} Nov 03 20:30:04.271818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.271800 4711 scope.go:122] "RemoveContainer" containerID="2fe82b01fad366b272884022a3af6b376a531b125295768cf2714ec27f1ecfd5" Nov 03 20:30:04.272159 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:04.272138 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j_odh-ai-gateway-infra(e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" Nov 03 20:30:04.273569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.273550 4711 generic.go:356] "Generic (PLEG): container finished" podID="4b295009-4e05-4150-b515-9c04a0040631" containerID="292f3cca562161110f3c233a32290b7ed5727d9399df2e183654560435a1a921" exitCode=60 Nov 03 20:30:04.273653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.273633 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" event={"ID":"4b295009-4e05-4150-b515-9c04a0040631","Type":"ContainerDied","Data":"292f3cca562161110f3c233a32290b7ed5727d9399df2e183654560435a1a921"} Nov 03 20:30:04.273918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.273902 4711 scope.go:122] "RemoveContainer" containerID="292f3cca562161110f3c233a32290b7ed5727d9399df2e183654560435a1a921" Nov 03 20:30:04.274253 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:04.274234 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs_odh-ai-gateway-infra(4b295009-4e05-4150-b515-9c04a0040631)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" podUID="4b295009-4e05-4150-b515-9c04a0040631" Nov 03 20:30:04.325435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.325413 4711 scope.go:122] "RemoveContainer" containerID="172190ed008f063067817f3682e0fad0fb7053f3c6fa6104dd456b915f0b63ff" Nov 03 20:30:04.340717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.340687 4711 scope.go:122] "RemoveContainer" containerID="6fdde45643dfde1d0b4fdf77e817b46502ed604bb547ab652c912a28a5518e28" Nov 03 20:30:04.356003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.355985 4711 scope.go:122] "RemoveContainer" containerID="2017efa9c1493a0eb4cbfc69383934291ae3e15d0b983ea97bdf8c16d3a6f623" Nov 03 20:30:04.640469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:04.637027 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:05.284663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:05.284623 4711 scope.go:122] "RemoveContainer" containerID="ada189c43c37fb5bcec9d11fd4294ec1af80dc9a251a31ebbb62509f18cf18f1" Nov 03 20:30:05.285170 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:05.285096 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k_odh-ai-gateway-infra(66805357-9d48-4395-b3ab-ae68309d7830)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" podUID="66805357-9d48-4395-b3ab-ae68309d7830" Nov 03 20:30:05.286516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:05.286463 4711 scope.go:122] "RemoveContainer" containerID="a85ea05ecb972081dcdcced656f694400499c72c38afdb2305be4adb0e72f95c" Nov 03 20:30:05.286949 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:05.286913 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j_odh-ai-gateway-infra(6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" Nov 03 20:30:05.288242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:05.288223 4711 scope.go:122] "RemoveContainer" containerID="2fe82b01fad366b272884022a3af6b376a531b125295768cf2714ec27f1ecfd5" Nov 03 20:30:05.288640 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:05.288618 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j_odh-ai-gateway-infra(e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" Nov 03 20:30:05.289940 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:05.289923 4711 scope.go:122] "RemoveContainer" containerID="292f3cca562161110f3c233a32290b7ed5727d9399df2e183654560435a1a921" Nov 03 20:30:05.290241 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:05.290223 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs_odh-ai-gateway-infra(4b295009-4e05-4150-b515-9c04a0040631)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" podUID="4b295009-4e05-4150-b515-9c04a0040631" Nov 03 20:30:05.332785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:05.332750 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:06.116884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:06.116846 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:06.818501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:06.818466 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:07.520604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:07.520571 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:08.215303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:08.215262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:08.915476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:08.910628 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:09.589062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:09.589023 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:10.363521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:10.363483 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:11.020102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:11.020060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:11.699097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:11.699064 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:12.455180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:12.455136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:13.325868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:13.325832 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:13.672405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:13.672325 4711 scope.go:122] "RemoveContainer" containerID="2fe82b01fad366b272884022a3af6b376a531b125295768cf2714ec27f1ecfd5" Nov 03 20:30:14.150248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:14.150209 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:14.412542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:14.412429 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" event={"ID":"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8","Type":"ContainerStarted","Data":"29bf16f8128dda1458291139dcf9e356be1d2e279b571078a1234ce4533fec08"} Nov 03 20:30:14.888152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:14.888113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:15.421303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:15.421269 4711 generic.go:356] "Generic (PLEG): container finished" podID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerID="29bf16f8128dda1458291139dcf9e356be1d2e279b571078a1234ce4533fec08" exitCode=60 Nov 03 20:30:15.421535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:15.421339 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" event={"ID":"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8","Type":"ContainerDied","Data":"29bf16f8128dda1458291139dcf9e356be1d2e279b571078a1234ce4533fec08"} Nov 03 20:30:15.421535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:15.421402 4711 scope.go:122] "RemoveContainer" containerID="2fe82b01fad366b272884022a3af6b376a531b125295768cf2714ec27f1ecfd5" Nov 03 20:30:15.421911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:15.421895 4711 scope.go:122] "RemoveContainer" containerID="29bf16f8128dda1458291139dcf9e356be1d2e279b571078a1234ce4533fec08" Nov 03 20:30:15.422794 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:15.422361 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j_odh-ai-gateway-infra(e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" Nov 03 20:30:15.442391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:15.442365 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j"] Nov 03 20:30:15.672781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:15.672695 4711 scope.go:122] "RemoveContainer" containerID="ada189c43c37fb5bcec9d11fd4294ec1af80dc9a251a31ebbb62509f18cf18f1" Nov 03 20:30:15.713701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:15.713662 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:16.430925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.430831 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" event={"ID":"66805357-9d48-4395-b3ab-ae68309d7830","Type":"ContainerStarted","Data":"644d8095ac826cc21bf900e68fef3ae9467835dfc92e8bdd6e12a2b1483b4377"} Nov 03 20:30:16.522075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.521929 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:16.663720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.663689 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:16.673617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.673593 4711 scope.go:122] "RemoveContainer" containerID="292f3cca562161110f3c233a32290b7ed5727d9399df2e183654560435a1a921" Nov 03 20:30:16.674773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.674510 4711 scope.go:122] "RemoveContainer" containerID="a85ea05ecb972081dcdcced656f694400499c72c38afdb2305be4adb0e72f95c" Nov 03 20:30:16.796773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.796737 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-maas-api-service-ca\") pod \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\" (UID: \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\") " Nov 03 20:30:16.797046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.797027 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-kube-api-access-27mdf\" (UniqueName: \"kubernetes.io/projected/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-kube-api-access-27mdf\") pod \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\" (UID: \"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8\") " Nov 03 20:30:16.797184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.797052 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-maas-api-service-ca" pod "e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" (UID: "e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:16.797902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.797883 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:16.799004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.798984 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-kube-api-access-27mdf" pod "e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" (UID: "e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8"). InnerVolumeSpecName "kube-api-access-27mdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:16.899897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:16.899868 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27mdf\" (UniqueName: \"kubernetes.io/projected/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8-kube-api-access-27mdf\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:17.369239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.369154 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:17.455413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.455381 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" event={"ID":"4b295009-4e05-4150-b515-9c04a0040631","Type":"ContainerStarted","Data":"ae9e59d5f6ac6bbaae1b6d50492762cbfb428dce130621c69ee39fc709bcefb2"} Nov 03 20:30:17.458050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.458020 4711 generic.go:356] "Generic (PLEG): container finished" podID="66805357-9d48-4395-b3ab-ae68309d7830" containerID="644d8095ac826cc21bf900e68fef3ae9467835dfc92e8bdd6e12a2b1483b4377" exitCode=60 Nov 03 20:30:17.458205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.458129 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" event={"ID":"66805357-9d48-4395-b3ab-ae68309d7830","Type":"ContainerDied","Data":"644d8095ac826cc21bf900e68fef3ae9467835dfc92e8bdd6e12a2b1483b4377"} Nov 03 20:30:17.458205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.458183 4711 scope.go:122] "RemoveContainer" containerID="ada189c43c37fb5bcec9d11fd4294ec1af80dc9a251a31ebbb62509f18cf18f1" Nov 03 20:30:17.458633 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.458611 4711 scope.go:122] "RemoveContainer" containerID="644d8095ac826cc21bf900e68fef3ae9467835dfc92e8bdd6e12a2b1483b4377" Nov 03 20:30:17.462645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.462582 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" event={"ID":"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665","Type":"ContainerStarted","Data":"808933206b96269c1ae5409d8b01f171c0aaf3ef070bfdc1eab4f792aaa6790d"} Nov 03 20:30:17.464733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.464706 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" event={"ID":"e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8","Type":"ContainerDied","Data":"bc7403d4e7574c36220a7f2d484d9a882398ef4becc07787c73e5ee72e556410"} Nov 03 20:30:17.464915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.464797 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j" Nov 03 20:30:17.468411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.466371 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k"] Nov 03 20:30:17.471153 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:17.470625 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k_odh-ai-gateway-infra(66805357-9d48-4395-b3ab-ae68309d7830)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" podUID="66805357-9d48-4395-b3ab-ae68309d7830" Nov 03 20:30:17.501136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.501107 4711 scope.go:122] "RemoveContainer" containerID="29bf16f8128dda1458291139dcf9e356be1d2e279b571078a1234ce4533fec08" Nov 03 20:30:17.535623 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:17.535592 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:30:17.546042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.546013 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j"] Nov 03 20:30:17.552381 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:17.552355 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-30e9de-29895630-hzg5j"] Nov 03 20:30:18.203897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.203822 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:18.488029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.487993 4711 generic.go:356] "Generic (PLEG): container finished" podID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerID="808933206b96269c1ae5409d8b01f171c0aaf3ef070bfdc1eab4f792aaa6790d" exitCode=60 Nov 03 20:30:18.488409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.488059 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" event={"ID":"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665","Type":"ContainerDied","Data":"808933206b96269c1ae5409d8b01f171c0aaf3ef070bfdc1eab4f792aaa6790d"} Nov 03 20:30:18.488409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.488165 4711 scope.go:122] "RemoveContainer" containerID="a85ea05ecb972081dcdcced656f694400499c72c38afdb2305be4adb0e72f95c" Nov 03 20:30:18.488603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.488587 4711 scope.go:122] "RemoveContainer" containerID="808933206b96269c1ae5409d8b01f171c0aaf3ef070bfdc1eab4f792aaa6790d" Nov 03 20:30:18.489059 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:18.489027 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j_odh-ai-gateway-infra(6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" Nov 03 20:30:18.491413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.491393 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs"] Nov 03 20:30:18.491631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.491609 4711 generic.go:356] "Generic (PLEG): container finished" podID="4b295009-4e05-4150-b515-9c04a0040631" containerID="ae9e59d5f6ac6bbaae1b6d50492762cbfb428dce130621c69ee39fc709bcefb2" exitCode=60 Nov 03 20:30:18.491751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.491630 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" event={"ID":"4b295009-4e05-4150-b515-9c04a0040631","Type":"ContainerDied","Data":"ae9e59d5f6ac6bbaae1b6d50492762cbfb428dce130621c69ee39fc709bcefb2"} Nov 03 20:30:18.510213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.510185 4711 scope.go:122] "RemoveContainer" containerID="292f3cca562161110f3c233a32290b7ed5727d9399df2e183654560435a1a921" Nov 03 20:30:18.526529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.526436 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j"] Nov 03 20:30:18.689113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.689080 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" path="/var/lib/kubelet/pods/e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8/volumes" Nov 03 20:30:18.691384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.691362 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:18.725196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.725167 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:18.843318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.843283 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b295009-4e05-4150-b515-9c04a0040631-kube-api-access-w4bts\" (UniqueName: \"kubernetes.io/projected/4b295009-4e05-4150-b515-9c04a0040631-kube-api-access-w4bts\") pod \"4b295009-4e05-4150-b515-9c04a0040631\" (UID: \"4b295009-4e05-4150-b515-9c04a0040631\") " Nov 03 20:30:18.843318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.843322 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b295009-4e05-4150-b515-9c04a0040631-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b295009-4e05-4150-b515-9c04a0040631-maas-api-service-ca\") pod \"4b295009-4e05-4150-b515-9c04a0040631\" (UID: \"4b295009-4e05-4150-b515-9c04a0040631\") " Nov 03 20:30:18.843634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.843456 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66805357-9d48-4395-b3ab-ae68309d7830-kube-api-access-tlgd2\" (UniqueName: \"kubernetes.io/projected/66805357-9d48-4395-b3ab-ae68309d7830-kube-api-access-tlgd2\") pod \"66805357-9d48-4395-b3ab-ae68309d7830\" (UID: \"66805357-9d48-4395-b3ab-ae68309d7830\") " Nov 03 20:30:18.843634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.843579 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66805357-9d48-4395-b3ab-ae68309d7830-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66805357-9d48-4395-b3ab-ae68309d7830-maas-api-service-ca\") pod \"66805357-9d48-4395-b3ab-ae68309d7830\" (UID: \"66805357-9d48-4395-b3ab-ae68309d7830\") " Nov 03 20:30:18.843875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.843845 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b295009-4e05-4150-b515-9c04a0040631-maas-api-service-ca" pod "4b295009-4e05-4150-b515-9c04a0040631" (UID: "4b295009-4e05-4150-b515-9c04a0040631"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:18.844244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.843982 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66805357-9d48-4395-b3ab-ae68309d7830-maas-api-service-ca" pod "66805357-9d48-4395-b3ab-ae68309d7830" (UID: "66805357-9d48-4395-b3ab-ae68309d7830"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:18.844816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.844795 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66805357-9d48-4395-b3ab-ae68309d7830-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.844938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.844814 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b295009-4e05-4150-b515-9c04a0040631-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.846144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.846119 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b295009-4e05-4150-b515-9c04a0040631-kube-api-access-w4bts" pod "4b295009-4e05-4150-b515-9c04a0040631" (UID: "4b295009-4e05-4150-b515-9c04a0040631"). InnerVolumeSpecName "kube-api-access-w4bts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:18.846293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.846280 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66805357-9d48-4395-b3ab-ae68309d7830-kube-api-access-tlgd2" pod "66805357-9d48-4395-b3ab-ae68309d7830" (UID: "66805357-9d48-4395-b3ab-ae68309d7830"). InnerVolumeSpecName "kube-api-access-tlgd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:18.957468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.946769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:18.957468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.948761 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlgd2\" (UniqueName: \"kubernetes.io/projected/66805357-9d48-4395-b3ab-ae68309d7830-kube-api-access-tlgd2\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.957468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:18.948798 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4bts\" (UniqueName: \"kubernetes.io/projected/4b295009-4e05-4150-b515-9c04a0040631-kube-api-access-w4bts\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.499794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.499763 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" Nov 03 20:30:19.500224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.499764 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k" event={"ID":"66805357-9d48-4395-b3ab-ae68309d7830","Type":"ContainerDied","Data":"2151006f227bb3871064392845c44f77fe076f77c6395c330571cb7a8fa1b6b7"} Nov 03 20:30:19.500224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.499960 4711 scope.go:122] "RemoveContainer" containerID="644d8095ac826cc21bf900e68fef3ae9467835dfc92e8bdd6e12a2b1483b4377" Nov 03 20:30:19.503112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.503086 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" Nov 03 20:30:19.503217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.503122 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs" event={"ID":"4b295009-4e05-4150-b515-9c04a0040631","Type":"ContainerDied","Data":"589cfdbbaab719e1714738696d1cd42d84c7edf112252174ad55ee928b810d90"} Nov 03 20:30:19.514223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.514205 4711 scope.go:122] "RemoveContainer" containerID="ae9e59d5f6ac6bbaae1b6d50492762cbfb428dce130621c69ee39fc709bcefb2" Nov 03 20:30:19.552937 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.552902 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs"] Nov 03 20:30:19.557163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.557137 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895630-9ldrs"] Nov 03 20:30:19.570648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.570614 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k"] Nov 03 20:30:19.580166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.579612 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895630-t895k"] Nov 03 20:30:19.642475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.637990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:19.750071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.749994 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:19.877353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.877322 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-maas-api-service-ca\") pod \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\" (UID: \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\") " Nov 03 20:30:19.877568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.877469 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-kube-api-access-dkrpj\" (UniqueName: \"kubernetes.io/projected/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-kube-api-access-dkrpj\") pod \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\" (UID: \"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665\") " Nov 03 20:30:19.877792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.877766 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-maas-api-service-ca" pod "6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" (UID: "6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:19.878154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.878136 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.879645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.879627 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-kube-api-access-dkrpj" pod "6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" (UID: "6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665"). InnerVolumeSpecName "kube-api-access-dkrpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:19.980266 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:19.980228 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkrpj\" (UniqueName: \"kubernetes.io/projected/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665-kube-api-access-dkrpj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.429427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:20.429385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:20.525813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:20.523751 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" event={"ID":"6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665","Type":"ContainerDied","Data":"041cf0e3ba41de738a2e964032972075ac1d9ea5b31d7dd04a6a55743595f023"} Nov 03 20:30:20.525813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:20.523803 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j" Nov 03 20:30:20.525813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:20.523846 4711 scope.go:122] "RemoveContainer" containerID="808933206b96269c1ae5409d8b01f171c0aaf3ef070bfdc1eab4f792aaa6790d" Nov 03 20:30:20.562124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:20.562090 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j"] Nov 03 20:30:20.569013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:20.568976 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895630-w856j"] Nov 03 20:30:20.682584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:20.682502 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b295009-4e05-4150-b515-9c04a0040631" path="/var/lib/kubelet/pods/4b295009-4e05-4150-b515-9c04a0040631/volumes" Nov 03 20:30:20.683070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:20.683048 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66805357-9d48-4395-b3ab-ae68309d7830" path="/var/lib/kubelet/pods/66805357-9d48-4395-b3ab-ae68309d7830/volumes" Nov 03 20:30:20.683548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:20.683530 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" path="/var/lib/kubelet/pods/6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665/volumes" Nov 03 20:30:21.232555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:21.232523 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:21.962842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:21.962808 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:22.661976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:22.635115 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:23.428469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:23.425005 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:24.059792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:24.059744 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:24.723238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:24.723205 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:25.423736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:25.423692 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:26.108648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:26.108605 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:26.933574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:26.933543 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:27.767467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:27.757246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:29.720062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:29.720025 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:30.954319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:30.954279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:31.798969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:31.798935 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:32.510801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:32.510721 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:32.920785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:32.920703 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:33.214387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:33.198184 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:33.913920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:33.913884 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:34.626035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:34.625988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:35.315457 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:35.315390 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:36.029102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:36.029063 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:36.701164 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:36.701129 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:37.446530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:37.446489 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:37.970933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:37.970896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:38.320323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:38.320287 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:39.011139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:39.011101 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:39.799721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:39.799684 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:40.619466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:40.609086 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:41.416910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:41.416868 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:42.194964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:42.194927 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:43.029924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:43.025213 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:43.802912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:43.802876 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:44.651626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:44.651592 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:45.391826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:45.391783 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:46.228678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:46.228643 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:46.915076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:46.915042 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:47.286354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:47.286319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:47.536190 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:30:47.536154 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:30:47.800454 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:47.800420 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:48.485179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.485146 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79"] Nov 03 20:30:48.486237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486208 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerName="cleanup" Nov 03 20:30:48.486237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486236 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486252 4711 cpu_manager.go:414] "containerMap: removing container" podUID="66805357-9d48-4395-b3ab-ae68309d7830" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486260 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66805357-9d48-4395-b3ab-ae68309d7830" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486282 4711 cpu_manager.go:414] "containerMap: removing container" podUID="66805357-9d48-4395-b3ab-ae68309d7830" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486289 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66805357-9d48-4395-b3ab-ae68309d7830" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486314 4711 cpu_manager.go:414] "containerMap: removing container" podUID="66805357-9d48-4395-b3ab-ae68309d7830" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486322 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66805357-9d48-4395-b3ab-ae68309d7830" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486365 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4b295009-4e05-4150-b515-9c04a0040631" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486372 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b295009-4e05-4150-b515-9c04a0040631" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486381 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4b295009-4e05-4150-b515-9c04a0040631" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486389 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b295009-4e05-4150-b515-9c04a0040631" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486412 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486419 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486430 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerName="cleanup" Nov 03 20:30:48.486434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486437 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486467 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486474 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486514 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486521 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486533 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486541 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486714 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="66805357-9d48-4395-b3ab-ae68309d7830" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486736 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b295009-4e05-4150-b515-9c04a0040631" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486743 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b295009-4e05-4150-b515-9c04a0040631" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486756 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486764 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486773 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486787 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486797 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f0e4f8-e699-4d18-8cab-7ebad68c4ee8" containerName="cleanup" Nov 03 20:30:48.487155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.486807 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="66805357-9d48-4395-b3ab-ae68309d7830" containerName="cleanup" Nov 03 20:30:48.492660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.492639 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.506750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.506717 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79"] Nov 03 20:30:48.606974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.606944 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.606974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.606975 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.607218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.607142 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.607218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.607180 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.607364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.607345 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.607411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.607380 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.607499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.607410 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.607546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.607508 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtlnb\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-kube-api-access-vtlnb\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.607582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.607565 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.708922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.708887 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.708934 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtlnb\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-kube-api-access-vtlnb\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.708999 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709069 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709092 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709167 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709191 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709249 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-data\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709376 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709454 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709420 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709524 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-credential-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709657 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-socket\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709794 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-certs\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.709877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.709852 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istiod-ca-cert\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.711419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.711392 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-envoy\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.711648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.711631 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-podinfo\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.726153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.726128 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtlnb\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-kube-api-access-vtlnb\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.727084 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.727054 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-token\") pod \"e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:48.834049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.834011 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:48.837585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:48.834598 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:49.171283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:49.171257 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79"] Nov 03 20:30:49.171417 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:30:49.171354 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c8e159_b4d1_4e4c_a25d_0be8fc3ec179.slice/crio-a6f4748e77529a8c4cc35a051a375a25d35bcd9a37269d27968fd29ce4b4102e WatchSource:0}: Error finding container a6f4748e77529a8c4cc35a051a375a25d35bcd9a37269d27968fd29ce4b4102e: Status 404 returned error can't find the container with id a6f4748e77529a8c4cc35a051a375a25d35bcd9a37269d27968fd29ce4b4102e Nov 03 20:30:49.174054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:49.174021 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:30:49.174127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:49.174086 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:30:49.174127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:49.174115 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:30:49.804882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:49.804840 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:49.915560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:49.915522 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" event={"ID":"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179","Type":"ContainerStarted","Data":"703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a"} Nov 03 20:30:49.915737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:49.915588 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" event={"ID":"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179","Type":"ContainerStarted","Data":"a6f4748e77529a8c4cc35a051a375a25d35bcd9a37269d27968fd29ce4b4102e"} Nov 03 20:30:49.946323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:49.946271 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" podStartSLOduration=1.946255533 podStartE2EDuration="1.946255533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:49.942312644 +0000 UTC m=+1753.392418690" watchObservedRunningTime="2026-11-03 20:30:49.946255533 +0000 UTC m=+1753.396361578" Nov 03 20:30:50.838721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:50.838688 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:50.840455 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:50.840414 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:30:50.840583 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:50.840480 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 20:30:51.408534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:51.408494 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:51.823466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:51.818331 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:51.840474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:51.838021 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:30:51.840474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:51.838086 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 20:30:52.516791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:52.516756 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:52.838197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:52.838111 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:30:52.838197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:52.838181 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 20:30:53.347211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:53.347179 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:53.838786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:53.838755 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:30:53.838981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:53.838815 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 20:30:54.212347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:54.212248 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:54.838490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:54.838436 4711 patch_prober.go:29] interesting pod/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:30:54.838975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:54.838517 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 03 20:30:55.101229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:55.101126 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:55.761916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:55.761880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:55.852330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:55.852304 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:55.856386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:55.856363 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:55.857399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:55.857377 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:30:56.583409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:56.583366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:57.319594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:57.319558 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:57.455739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:57.455688 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:58.008366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:58.008332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:58.690384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:58.690351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:59.371787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:30:59.371743 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:00.039792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:00.039759 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:00.796119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:00.796078 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:01.497484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:01.497427 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:02.232621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:02.232550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:02.910580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:02.910520 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:03.589822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:03.589754 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:04.309675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:04.309647 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:05.104081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:05.104048 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:05.895185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:05.895134 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:07.091573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:07.091523 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:07.888352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:07.888313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:08.748422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:08.747050 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:09.453261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:09.453227 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:10.122793 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:10.122759 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:10.803723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:10.803681 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:11.516763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:11.516717 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:12.229683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:12.229630 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:12.990826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:12.990791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:13.826343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:13.826310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:14.515194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:14.515150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:14.678469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:14.671426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:15.058179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:15.058143 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:15.273468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:15.262330 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:15.816605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:15.816562 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:15.996268 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:15.996236 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:16.805466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:16.802075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:17.533785 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:31:17.533742 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:31:17.621754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:17.621717 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:18.363507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:18.363473 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:19.254847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:19.249341 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:20.110921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:20.110883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:20.928826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:20.928785 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:21.700593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:21.700556 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:21.835837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:21.835803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:22.494842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:22.494803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:23.230352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:23.230311 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:23.920995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:23.920945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:24.715745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:24.715709 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:25.550012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:25.547838 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:26.242468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:26.240031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:26.856282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:26.856248 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:27.182963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:27.182870 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:27.532208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:27.532174 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:28.284525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:28.284483 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:29.943129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:29.943096 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:30.793066 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:30.792998 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:31.588756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:31.588710 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:31.950590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:31.932989 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:32.553269 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:32.553234 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:33.198655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:33.198616 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:33.893511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:33.893476 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:34.697374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:34.697340 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:35.405355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:35.405319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:36.103895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:36.103857 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:36.781845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:36.781808 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:36.968756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:36.968714 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:37.165226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:37.165147 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:31:37.169605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:37.169582 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:31:37.175329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:37.175308 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:31:37.179004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:37.178979 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:31:37.615987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:37.615951 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:38.503833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:38.503799 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:39.145922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:39.145891 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:39.684519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:39.684435 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:40.439802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:40.439735 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:41.193838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:41.193797 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:41.989569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:41.989528 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:42.814325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:42.814290 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:43.549245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:43.549190 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:43.699065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:43.699022 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:43.832999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:43.832914 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:44.198973 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.198882 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:44.514560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.514521 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m"] Nov 03 20:31:44.514812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.514783 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" containerID="cri-o://7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c" gracePeriod=30 Nov 03 20:31:44.520684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.520657 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c"] Nov 03 20:31:44.521265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.521226 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" containerID="cri-o://219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067" gracePeriod=30 Nov 03 20:31:44.536186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.536162 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29"] Nov 03 20:31:44.536730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.536698 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" containerID="cri-o://592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea" gracePeriod=30 Nov 03 20:31:44.538728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.538704 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79"] Nov 03 20:31:44.538942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.538917 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerName="istio-proxy" containerID="cri-o://703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a" gracePeriod=30 Nov 03 20:31:44.653127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.653088 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:44.791490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:44.791387 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:45.383884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:45.383841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:45.922750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:45.922707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:46.711667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:46.711553 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:47.535669 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:31:47.535630 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:31:47.587772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:47.587724 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:48.312768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:48.312726 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:49.020651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:49.002407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:49.685282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:49.684662 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:49.824571 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:49.823062 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:49.945663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:49.945635 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:31:49.999617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:49.999587 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:31:50.005174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.004694 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:31:50.009072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.009050 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:31:50.095620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.095539 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-credential-socket\") pod \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " Nov 03 20:31:50.095873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.095853 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-data\") pod \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " Nov 03 20:31:50.096028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.096006 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-credential-socket" pod "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" (UID: "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.096106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.096019 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istiod-ca-cert\") pod \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " Nov 03 20:31:50.096312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.096265 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-data" pod "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" (UID: "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.098789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.096428 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istiod-ca-cert" pod "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" (UID: "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:50.098789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.096332 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-token\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-token\") pod \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " Nov 03 20:31:50.098949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.098919 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-token\") pod \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.098991 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-socket\") pod \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099070 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istiod-ca-cert\") pod \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099115 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-podinfo\") pod \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099167 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-podinfo\") pod \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099222 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-certs\") pod \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099254 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-data\") pod \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099309 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-certs\") pod \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099338 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-podinfo\") pod \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099395 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-token" pod "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" (UID: "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099411 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-credential-socket\") pod \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099474 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-kube-api-access-vtlnb\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-kube-api-access-vtlnb\") pod \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099512 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-envoy\") pod \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099584 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-certs\") pod \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099642 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-credential-socket\") pod \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099674 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-socket\") pod \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " Nov 03 20:31:50.099832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099816 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-envoy\") pod \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " Nov 03 20:31:50.100820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.099967 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-credential-socket\") pod \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " Nov 03 20:31:50.100820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.100304 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-socket" pod "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" (UID: "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.100820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.100730 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istiod-ca-cert" pod "a3ad1145-1b5a-43db-ae2d-704d6e113f61" (UID: "a3ad1145-1b5a-43db-ae2d-704d6e113f61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:50.101051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.101031 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-podinfo\") pod \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " Nov 03 20:31:50.101125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.101106 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-certs\") pod \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\" (UID: \"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179\") " Nov 03 20:31:50.101186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.101145 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istiod-ca-cert\") pod \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " Nov 03 20:31:50.101751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.101727 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-podinfo" pod "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" (UID: "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:50.101973 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.101952 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-certs" pod "9a228a8b-66b7-4c86-ac95-c24bb5f3043c" (UID: "9a228a8b-66b7-4c86-ac95-c24bb5f3043c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.102215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102194 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-certs" pod "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" (UID: "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.102482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102423 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.102482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102458 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.102482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102473 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.102680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102487 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.102680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102500 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.102680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102514 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.102680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102527 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.102888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102714 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-data" pod "9a228a8b-66b7-4c86-ac95-c24bb5f3043c" (UID: "9a228a8b-66b7-4c86-ac95-c24bb5f3043c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.102943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.102894 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-credential-socket" pod "a3ad1145-1b5a-43db-ae2d-704d6e113f61" (UID: "a3ad1145-1b5a-43db-ae2d-704d6e113f61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.103087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.103067 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-certs" pod "a3ad1145-1b5a-43db-ae2d-704d6e113f61" (UID: "a3ad1145-1b5a-43db-ae2d-704d6e113f61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.103273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.103252 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-credential-socket" pod "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" (UID: "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.103463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.103429 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-socket" pod "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" (UID: "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.104205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.103965 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-credential-socket" pod "9a228a8b-66b7-4c86-ac95-c24bb5f3043c" (UID: "9a228a8b-66b7-4c86-ac95-c24bb5f3043c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.104205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.104172 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-certs" pod "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" (UID: "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.104358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.104241 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istiod-ca-cert" pod "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" (UID: "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:50.106277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.106255 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-podinfo" pod "9a228a8b-66b7-4c86-ac95-c24bb5f3043c" (UID: "9a228a8b-66b7-4c86-ac95-c24bb5f3043c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:50.108322 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.108301 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-envoy" pod "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" (UID: "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.108893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.108849 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-envoy" pod "a3ad1145-1b5a-43db-ae2d-704d6e113f61" (UID: "a3ad1145-1b5a-43db-ae2d-704d6e113f61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.109112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.109083 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-podinfo" pod "a3ad1145-1b5a-43db-ae2d-704d6e113f61" (UID: "a3ad1145-1b5a-43db-ae2d-704d6e113f61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:50.113831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.113813 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-podinfo" pod "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" (UID: "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:50.113914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.113847 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-token" pod "a3ad1145-1b5a-43db-ae2d-704d6e113f61" (UID: "a3ad1145-1b5a-43db-ae2d-704d6e113f61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.115398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.115374 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-kube-api-access-vtlnb" pod "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" (UID: "c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179"). InnerVolumeSpecName "kube-api-access-vtlnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.203269 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203234 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-data\") pod \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " Nov 03 20:31:50.203428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203291 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-socket\") pod \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " Nov 03 20:31:50.203428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203346 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-token\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-token\") pod \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " Nov 03 20:31:50.203428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203386 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-envoy\") pod \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " Nov 03 20:31:50.203428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203418 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-kube-api-access-hmr2m\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-kube-api-access-hmr2m\") pod \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " Nov 03 20:31:50.203689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203462 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-kube-api-access-c27pd\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-kube-api-access-c27pd\") pod \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " Nov 03 20:31:50.203689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203494 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-token\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-token\") pod \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " Nov 03 20:31:50.203689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203505 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-data" pod "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" (UID: "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.203689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203640 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-socket" pod "a3ad1145-1b5a-43db-ae2d-704d6e113f61" (UID: "a3ad1145-1b5a-43db-ae2d-704d6e113f61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.203910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203873 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-socket\") pod \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " Nov 03 20:31:50.203972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203913 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-kube-api-access-qb2lr\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-kube-api-access-qb2lr\") pod \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " Nov 03 20:31:50.203972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.203943 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-data\") pod \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\" (UID: \"a3ad1145-1b5a-43db-ae2d-704d6e113f61\") " Nov 03 20:31:50.204080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.204005 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-envoy\") pod \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\" (UID: \"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf\") " Nov 03 20:31:50.204140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.204130 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istiod-ca-cert\") pod \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\" (UID: \"9a228a8b-66b7-4c86-ac95-c24bb5f3043c\") " Nov 03 20:31:50.204580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.204406 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-socket" pod "9a228a8b-66b7-4c86-ac95-c24bb5f3043c" (UID: "9a228a8b-66b7-4c86-ac95-c24bb5f3043c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.204710 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istiod-ca-cert" pod "9a228a8b-66b7-4c86-ac95-c24bb5f3043c" (UID: "9a228a8b-66b7-4c86-ac95-c24bb5f3043c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.204752 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-data" pod "a3ad1145-1b5a-43db-ae2d-704d6e113f61" (UID: "a3ad1145-1b5a-43db-ae2d-704d6e113f61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205008 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205034 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205052 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205067 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205081 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205096 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205111 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205126 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205142 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205157 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205170 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205185 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205201 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205214 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205230 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205244 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205259 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtlnb\" (UniqueName: \"kubernetes.io/projected/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-kube-api-access-vtlnb\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205274 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205291 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad1145-1b5a-43db-ae2d-704d6e113f61-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205305 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205320 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.205358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.205334 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.207077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.206980 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-token" pod "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" (UID: "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.207077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.207040 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-token" pod "9a228a8b-66b7-4c86-ac95-c24bb5f3043c" (UID: "9a228a8b-66b7-4c86-ac95-c24bb5f3043c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.207233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.207166 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-envoy" pod "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" (UID: "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.207366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.207341 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-kube-api-access-hmr2m" pod "a3ad1145-1b5a-43db-ae2d-704d6e113f61" (UID: "a3ad1145-1b5a-43db-ae2d-704d6e113f61"). InnerVolumeSpecName "kube-api-access-hmr2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.208434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.208409 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-kube-api-access-qb2lr" pod "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" (UID: "87b6a1c4-e8bc-44c8-8a59-bc02e35397cf"). InnerVolumeSpecName "kube-api-access-qb2lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.208599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.208415 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-kube-api-access-c27pd" pod "9a228a8b-66b7-4c86-ac95-c24bb5f3043c" (UID: "9a228a8b-66b7-4c86-ac95-c24bb5f3043c"). InnerVolumeSpecName "kube-api-access-c27pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:50.208781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.208754 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-envoy" pod "9a228a8b-66b7-4c86-ac95-c24bb5f3043c" (UID: "9a228a8b-66b7-4c86-ac95-c24bb5f3043c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:50.307067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.307030 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.307067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.307065 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmr2m\" (UniqueName: \"kubernetes.io/projected/a3ad1145-1b5a-43db-ae2d-704d6e113f61-kube-api-access-hmr2m\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.307289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.307078 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c27pd\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-kube-api-access-c27pd\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.307289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.307092 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.307289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.307104 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb2lr\" (UniqueName: \"kubernetes.io/projected/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-kube-api-access-qb2lr\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.307289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.307117 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.307289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.307128 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a228a8b-66b7-4c86-ac95-c24bb5f3043c-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:50.493576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.493477 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:50.559661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.559417 4711 generic.go:356] "Generic (PLEG): container finished" podID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerID="219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067" exitCode=0 Nov 03 20:31:50.559661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.559511 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" event={"ID":"9a228a8b-66b7-4c86-ac95-c24bb5f3043c","Type":"ContainerDied","Data":"219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067"} Nov 03 20:31:50.559661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.559594 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" event={"ID":"9a228a8b-66b7-4c86-ac95-c24bb5f3043c","Type":"ContainerDied","Data":"339229833f3716548caf61af2b9d14d161392498e87c6090cab94ce07c71c599"} Nov 03 20:31:50.559661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.559628 4711 scope.go:122] "RemoveContainer" containerID="219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067" Nov 03 20:31:50.560933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.559836 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c" Nov 03 20:31:50.569851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.562855 4711 generic.go:356] "Generic (PLEG): container finished" podID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerID="592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea" exitCode=0 Nov 03 20:31:50.569851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.562929 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" event={"ID":"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf","Type":"ContainerDied","Data":"592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea"} Nov 03 20:31:50.569851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.562962 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" event={"ID":"87b6a1c4-e8bc-44c8-8a59-bc02e35397cf","Type":"ContainerDied","Data":"6da250c15986c52e202e89fe33d0655269f6aefd240988eb7cbe4a7e4ae9de25"} Nov 03 20:31:50.569851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.563018 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29" Nov 03 20:31:50.569851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.565799 4711 generic.go:356] "Generic (PLEG): container finished" podID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerID="7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c" exitCode=0 Nov 03 20:31:50.569851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.565850 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" Nov 03 20:31:50.569851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.565962 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" event={"ID":"a3ad1145-1b5a-43db-ae2d-704d6e113f61","Type":"ContainerDied","Data":"7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c"} Nov 03 20:31:50.569851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.566001 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m" event={"ID":"a3ad1145-1b5a-43db-ae2d-704d6e113f61","Type":"ContainerDied","Data":"15a81f72fa860a5ffd9a3999f930017ad51eb4842a02e9825317c213e454dfce"} Nov 03 20:31:50.573058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.572966 4711 generic.go:356] "Generic (PLEG): container finished" podID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerID="703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a" exitCode=0 Nov 03 20:31:50.573058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.573034 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" Nov 03 20:31:50.573058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.573056 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" event={"ID":"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179","Type":"ContainerDied","Data":"703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a"} Nov 03 20:31:50.573313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.573089 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79" event={"ID":"c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179","Type":"ContainerDied","Data":"a6f4748e77529a8c4cc35a051a375a25d35bcd9a37269d27968fd29ce4b4102e"} Nov 03 20:31:50.606152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.604720 4711 scope.go:122] "RemoveContainer" containerID="219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067" Nov 03 20:31:50.606152 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:31:50.605148 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067\": container with ID starting with 219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067 not found: ID does not exist" containerID="219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067" Nov 03 20:31:50.606152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.605184 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067"} err="failed to get container status \"219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067\": rpc error: code = NotFound desc = could not find container \"219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067\": container with ID starting with 219bc302d4bf6ee28f925f866fd89ab94e84861fa356187563fc041080063067 not found: ID does not exist" Nov 03 20:31:50.606152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.605209 4711 scope.go:122] "RemoveContainer" containerID="592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea" Nov 03 20:31:50.617278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.617258 4711 scope.go:122] "RemoveContainer" containerID="592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea" Nov 03 20:31:50.617627 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:31:50.617607 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea\": container with ID starting with 592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea not found: ID does not exist" containerID="592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea" Nov 03 20:31:50.617713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.617635 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea"} err="failed to get container status \"592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea\": rpc error: code = NotFound desc = could not find container \"592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea\": container with ID starting with 592c85ced30d280c92b92f1a79a85e78674ee1e7f40bec56f17bbb39132da2ea not found: ID does not exist" Nov 03 20:31:50.617713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.617653 4711 scope.go:122] "RemoveContainer" containerID="7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c" Nov 03 20:31:50.627135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.627111 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m"] Nov 03 20:31:50.628302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.628280 4711 scope.go:122] "RemoveContainer" containerID="7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c" Nov 03 20:31:50.628637 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:31:50.628609 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c\": container with ID starting with 7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c not found: ID does not exist" containerID="7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c" Nov 03 20:31:50.628746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.628646 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c"} err="failed to get container status \"7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c\": rpc error: code = NotFound desc = could not find container \"7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c\": container with ID starting with 7b8f0ad7d05f0a7c83d86c871db72873cf4fd05877c3944913f74206fbeba58c not found: ID does not exist" Nov 03 20:31:50.628746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.628670 4711 scope.go:122] "RemoveContainer" containerID="703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a" Nov 03 20:31:50.631739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.631719 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-nks2m"] Nov 03 20:31:50.643540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.643507 4711 scope.go:122] "RemoveContainer" containerID="703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a" Nov 03 20:31:50.645260 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:31:50.645226 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a\": container with ID starting with 703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a not found: ID does not exist" containerID="703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a" Nov 03 20:31:50.645405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.645269 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a"} err="failed to get container status \"703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a\": rpc error: code = NotFound desc = could not find container \"703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a\": container with ID starting with 703971cd5f05dc7679e225f92a9ad1297cd5045183ec27546afe9807396a3e9a not found: ID does not exist" Nov 03 20:31:50.650401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.650380 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29"] Nov 03 20:31:50.653561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.653538 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:50.654650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.654631 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-t9b29"] Nov 03 20:31:50.661963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.661941 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c"] Nov 03 20:31:50.666487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.666460 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-d985c"] Nov 03 20:31:50.683623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.683593 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" path="/var/lib/kubelet/pods/87b6a1c4-e8bc-44c8-8a59-bc02e35397cf/volumes" Nov 03 20:31:50.684218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.684197 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" path="/var/lib/kubelet/pods/9a228a8b-66b7-4c86-ac95-c24bb5f3043c/volumes" Nov 03 20:31:50.684813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.684775 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" path="/var/lib/kubelet/pods/a3ad1145-1b5a-43db-ae2d-704d6e113f61/volumes" Nov 03 20:31:50.685476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.685440 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79"] Nov 03 20:31:50.685775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:50.685498 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3211a0-openshift-default-748fcc5f5c-c2m79"] Nov 03 20:31:51.132797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:51.132539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:51.593807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:51.593775 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:51.716990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:51.716953 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:52.153754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:52.153714 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:52.325078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:52.325037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:52.683273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:52.683238 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" path="/var/lib/kubelet/pods/c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179/volumes" Nov 03 20:31:53.170640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.170598 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w"] Nov 03 20:31:53.171263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.170854 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" podUID="53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" containerName="maas-api" containerID="cri-o://099ebb3797e2b1bc5275ef84a99af50bb4631a6c6f36be47fa52b77790e8381c" gracePeriod=30 Nov 03 20:31:53.251541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.251508 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:53.673309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.673269 4711 generic.go:356] "Generic (PLEG): container finished" podID="53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" containerID="099ebb3797e2b1bc5275ef84a99af50bb4631a6c6f36be47fa52b77790e8381c" exitCode=0 Nov 03 20:31:53.673594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.673565 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" event={"ID":"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9","Type":"ContainerDied","Data":"099ebb3797e2b1bc5275ef84a99af50bb4631a6c6f36be47fa52b77790e8381c"} Nov 03 20:31:53.673693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.673618 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" event={"ID":"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9","Type":"ContainerDied","Data":"57d33e30e2f98b77651bb15441988a9264e0fb203f857edf037dbce607618ecd"} Nov 03 20:31:53.673693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.673644 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57d33e30e2f98b77651bb15441988a9264e0fb203f857edf037dbce607618ecd" Nov 03 20:31:53.674315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.674290 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:31:53.773080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.773046 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-metrics-certs\") pod \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " Nov 03 20:31:53.773271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.773186 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-maas-api-tls\") pod \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " Nov 03 20:31:53.773323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.773273 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-openshift-service-ca\") pod \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " Nov 03 20:31:53.773373 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.773341 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-kube-api-access-4chtv\" (UniqueName: \"kubernetes.io/projected/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-kube-api-access-4chtv\") pod \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\" (UID: \"53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9\") " Nov 03 20:31:53.773709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.773669 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-openshift-service-ca" pod "53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" (UID: "53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:53.775276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.775252 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.775558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.775537 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-metrics-certs" pod "53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" (UID: "53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:53.775861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.775834 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-kube-api-access-4chtv" pod "53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" (UID: "53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9"). InnerVolumeSpecName "kube-api-access-4chtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:53.777558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.777534 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-maas-api-tls" pod "53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" (UID: "53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:53.877400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.877318 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.877400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.877350 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.877400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.877360 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4chtv\" (UniqueName: \"kubernetes.io/projected/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9-kube-api-access-4chtv\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:53.879090 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:53.879060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:54.285770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:54.284464 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:54.686370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:54.686283 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w" Nov 03 20:31:54.722267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:54.719165 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w"] Nov 03 20:31:54.723938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:54.723913 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6bd876-788bff7c4d-pzb2w"] Nov 03 20:31:54.727194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:54.727168 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:54.924222 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:54.924186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:55.135390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:55.135354 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:55.335114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:55.334576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:56.142864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:56.142826 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:56.270391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:56.270357 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:56.397968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:56.397881 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:56.692052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:56.691967 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" path="/var/lib/kubelet/pods/53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9/volumes" Nov 03 20:31:57.475469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:57.459225 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:57.604768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:57.604719 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:58.164178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:58.164140 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:58.299976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:58.299924 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:58.769737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:58.769702 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:58.904370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:58.904336 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:59.114819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:59.114722 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:59.279477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:59.278788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:59.418240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:59.418132 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:31:59.738320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:59.738233 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz"] Nov 03 20:31:59.738546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:59.738515 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" containerID="cri-o://66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b" gracePeriod=30 Nov 03 20:31:59.807132 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:31:59.807093 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:00.393051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:00.393013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:00.756774 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:00.756719 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:01.098405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:01.098322 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:01.249908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:01.249877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:01.385523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:01.385319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:01.791772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:01.791720 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:02.185490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:02.185356 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:02.328699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:02.328665 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:02.466085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:02.465990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:02.778239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:02.778207 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:02.983366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:02.983332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:03.415832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:03.415800 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:03.967332 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:03.967288 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:04.115932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:04.115894 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:05.030310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.030282 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:32:05.043669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.043642 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:05.063969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.063912 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-envoy\") pod \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " Nov 03 20:32:05.064511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.064485 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-certs\") pod \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " Nov 03 20:32:05.075923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.075888 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-token\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-token\") pod \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " Nov 03 20:32:05.076218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.076185 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-socket\") pod \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " Nov 03 20:32:05.076315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.076256 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-podinfo\") pod \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " Nov 03 20:32:05.076378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.076321 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-kube-api-access-qmhqh\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-kube-api-access-qmhqh\") pod \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " Nov 03 20:32:05.076440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.076387 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istiod-ca-cert\") pod \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " Nov 03 20:32:05.076440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.076421 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-data\") pod \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " Nov 03 20:32:05.076619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.076562 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-credential-socket\") pod \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\" (UID: \"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799\") " Nov 03 20:32:05.095464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.094767 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-credential-socket" pod "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" (UID: "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:05.108131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.107970 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-data" pod "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" (UID: "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:05.108131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.107983 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-certs" pod "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" (UID: "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:05.108355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.108240 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-socket" pod "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" (UID: "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:05.121313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.121275 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-token" pod "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" (UID: "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:05.124800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.124766 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istiod-ca-cert" pod "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" (UID: "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:05.125313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.125268 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-kube-api-access-qmhqh" pod "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" (UID: "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799"). InnerVolumeSpecName "kube-api-access-qmhqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:05.128403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.128375 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-podinfo" pod "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" (UID: "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:05.128943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.128908 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-envoy" pod "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" (UID: "7e00cfda-16b5-4c4f-b5fc-616c2c9ad799"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:05.180067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.180032 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:05.180067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.180070 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:05.180264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.180085 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmhqh\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-kube-api-access-qmhqh\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:05.180264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.180102 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:05.180264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.180115 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:05.180264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.180128 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:05.180264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.180142 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:05.180264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.180156 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:05.180264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.180169 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:05.419586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.419492 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:05.565472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.561125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:05.736904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.736809 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:05.793865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.793830 4711 generic.go:356] "Generic (PLEG): container finished" podID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerID="66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b" exitCode=0 Nov 03 20:32:05.794024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.793905 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" Nov 03 20:32:05.794024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.793907 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" event={"ID":"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799","Type":"ContainerDied","Data":"66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b"} Nov 03 20:32:05.794024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.794003 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz" event={"ID":"7e00cfda-16b5-4c4f-b5fc-616c2c9ad799","Type":"ContainerDied","Data":"54584f9b083abee6671d817de1312b9a7c16993193148b022c9d1d0425349cd0"} Nov 03 20:32:05.794189 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.794078 4711 scope.go:122] "RemoveContainer" containerID="66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b" Nov 03 20:32:05.805292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.805276 4711 scope.go:122] "RemoveContainer" containerID="4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9" Nov 03 20:32:05.816962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.816938 4711 scope.go:122] "RemoveContainer" containerID="66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b" Nov 03 20:32:05.817316 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:32:05.817292 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b\": container with ID starting with 66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b not found: ID does not exist" containerID="66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b" Nov 03 20:32:05.817411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.817329 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b"} err="failed to get container status \"66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b\": rpc error: code = NotFound desc = could not find container \"66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b\": container with ID starting with 66d57d89a523bfba4a98f5e80cd84e67b31c503dce5c9c58caf077e6813e3b1b not found: ID does not exist" Nov 03 20:32:05.817411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.817353 4711 scope.go:122] "RemoveContainer" containerID="4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9" Nov 03 20:32:05.817695 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:32:05.817676 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9\": container with ID starting with 4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9 not found: ID does not exist" containerID="4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9" Nov 03 20:32:05.817751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.817702 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9"} err="failed to get container status \"4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9\": rpc error: code = NotFound desc = could not find container \"4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9\": container with ID starting with 4c01a894c3ebc280560e84277aca7e2d301e500c73102f98c446f7c2b85187c9 not found: ID does not exist" Nov 03 20:32:05.820932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.820911 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz"] Nov 03 20:32:05.828611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.828591 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-6bd876-openshift-default-db6f74989-6rhwz"] Nov 03 20:32:05.873708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:05.873666 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:06.063018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:06.062727 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:06.685239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:06.685206 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" path="/var/lib/kubelet/pods/7e00cfda-16b5-4c4f-b5fc-616c2c9ad799/volumes" Nov 03 20:32:06.884464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:06.884413 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:07.018254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:07.018224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:07.274518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:07.268836 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:07.705797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:07.705708 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:07.860401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:07.860362 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:08.146310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:08.146272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:08.944051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:08.944020 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:09.077060 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:09.077029 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:09.312400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:09.310150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:09.445403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:09.445351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:10.202379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.202346 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d"] Nov 03 20:32:10.203418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.203393 4711 cpu_manager.go:414] "containerMap: removing container" podUID="53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" containerName="maas-api" Nov 03 20:32:10.203620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.203603 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" containerName="maas-api" Nov 03 20:32:10.203743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.203732 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" Nov 03 20:32:10.203837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.203826 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" Nov 03 20:32:10.203939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.203929 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4b295009-4e05-4150-b515-9c04a0040631" containerName="cleanup" Nov 03 20:32:10.204013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204004 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b295009-4e05-4150-b515-9c04a0040631" containerName="cleanup" Nov 03 20:32:10.204128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204111 4711 cpu_manager.go:414] "containerMap: removing container" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" Nov 03 20:32:10.204128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204124 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" Nov 03 20:32:10.204321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204161 4711 cpu_manager.go:414] "containerMap: removing container" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerName="istio-proxy" Nov 03 20:32:10.204321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204169 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerName="istio-proxy" Nov 03 20:32:10.204321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204184 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" Nov 03 20:32:10.204321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204191 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" Nov 03 20:32:10.204321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204237 4711 cpu_manager.go:414] "containerMap: removing container" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" Nov 03 20:32:10.204321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204244 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204460 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a228a8b-66b7-4c86-ac95-c24bb5f3043c" containerName="istio-proxy" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204472 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b295009-4e05-4150-b515-9c04a0040631" containerName="cleanup" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204487 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b6a1c4-e8bc-44c8-8a59-bc02e35397cf" containerName="istio-proxy" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204499 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6764eac5-a3cb-4b6c-8fd3-0e2c7d35f665" containerName="cleanup" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204509 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c8e159-b4d1-4e4c-a25d-0be8fc3ec179" containerName="istio-proxy" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204520 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204532 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204544 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ad1145-1b5a-43db-ae2d-704d6e113f61" containerName="istio-proxy" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204556 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="66805357-9d48-4395-b3ab-ae68309d7830" containerName="cleanup" Nov 03 20:32:10.204657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.204570 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b8e6f5-b1b7-4ab8-8a54-1f0613e559a9" containerName="maas-api" Nov 03 20:32:10.212209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.212182 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.217650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.217365 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d"] Nov 03 20:32:10.220915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.220891 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-dce009-serving-cert\"" Nov 03 20:32:10.287306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.287257 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:10.329116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.329076 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.329283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.329188 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5fc815-027b-480b-bcda-a79f4d1bacd5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.329283 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.329226 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-metrics-certs\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.329538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.329520 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6hpx\" (UniqueName: \"kubernetes.io/projected/6f5fc815-027b-480b-bcda-a79f4d1bacd5-kube-api-access-b6hpx\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.430463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.429334 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:10.430989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.430961 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6hpx\" (UniqueName: \"kubernetes.io/projected/6f5fc815-027b-480b-bcda-a79f4d1bacd5-kube-api-access-b6hpx\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.431095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.431067 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.432053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.431141 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5fc815-027b-480b-bcda-a79f4d1bacd5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.432053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.431188 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-metrics-certs\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.444807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.433085 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5fc815-027b-480b-bcda-a79f4d1bacd5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.476219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.466733 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6hpx\" (UniqueName: \"kubernetes.io/projected/6f5fc815-027b-480b-bcda-a79f4d1bacd5-kube-api-access-b6hpx\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.487504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.483876 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-metrics-certs\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.487504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.485178 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.536674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.536634 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:10.559999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.559922 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:10.947093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:10.947063 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d"] Nov 03 20:32:10.955484 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:32:10.949681 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5fc815_027b_480b_bcda_a79f4d1bacd5.slice/crio-f69a63fb660fa5859000287790ebb6e954a4c0a3f4b396175be0931b5e697cc6 WatchSource:0}: Error finding container f69a63fb660fa5859000287790ebb6e954a4c0a3f4b396175be0931b5e697cc6: Status 404 returned error can't find the container with id f69a63fb660fa5859000287790ebb6e954a4c0a3f4b396175be0931b5e697cc6 Nov 03 20:32:11.235418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:11.235334 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:11.912530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:11.912414 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" event={"ID":"6f5fc815-027b-480b-bcda-a79f4d1bacd5","Type":"ContainerStarted","Data":"e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479"} Nov 03 20:32:11.912530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:11.912499 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" event={"ID":"6f5fc815-027b-480b-bcda-a79f4d1bacd5","Type":"ContainerStarted","Data":"f69a63fb660fa5859000287790ebb6e954a4c0a3f4b396175be0931b5e697cc6"} Nov 03 20:32:11.912530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:11.912523 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:11.933846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:11.933795 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" podStartSLOduration=1.330675091 podStartE2EDuration="1.933780807s" totalImagesPullingTime="603.105716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:11.933189842 +0000 UTC m=+1835.383295890" watchObservedRunningTime="2026-11-03 20:32:11.933780807 +0000 UTC m=+1835.383886851" Nov 03 20:32:12.025584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:12.025554 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:13.016972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:13.016933 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:14.841722 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:14.841682 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:15.830059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:15.830026 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9"] Nov 03 20:32:15.830247 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:15.830210 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" containerID="cri-o://18305c90fd035110ad9f958b68ff2a84cd22753138d857681f773462927aad03" gracePeriod=30 Nov 03 20:32:15.933219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:15.933182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:16.440152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:16.440114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:16.584778 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:16.584732 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:17.089805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.089773 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc"] Nov 03 20:32:17.090589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.090573 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" Nov 03 20:32:17.090589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.090590 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e00cfda-16b5-4c4f-b5fc-616c2c9ad799" containerName="istio-proxy" Nov 03 20:32:17.094429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.094401 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.119262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.119235 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-db5198-openshift-default-dockercfg-wv2ss\"" Nov 03 20:32:17.120100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.120073 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc"] Nov 03 20:32:17.193521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.193490 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:17.220882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.220852 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4153e58d-b2c1-48c5-9175-984b7115edb5-istiod-ca-cert\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.220882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.220891 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-envoy\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.221124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.221047 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-credential-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.221124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.221104 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhwj\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-kube-api-access-7fhwj\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.221312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.221283 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.221524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.221498 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-data\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.221689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.221668 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-token\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.221767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.221756 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-podinfo\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.221895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.221874 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-certs\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.348536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.345283 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.348536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.345434 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-data\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.348536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.345526 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-token\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.348536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.345584 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-podinfo\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.348536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.345630 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-certs\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.348536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.345698 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4153e58d-b2c1-48c5-9175-984b7115edb5-istiod-ca-cert\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.348536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.345730 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-envoy\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.348536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.345818 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-credential-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.348536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.345865 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhwj\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-kube-api-access-7fhwj\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.389805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.389223 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4153e58d-b2c1-48c5-9175-984b7115edb5-istiod-ca-cert\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.401352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.396811 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.401352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.400801 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-data\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.450468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.449579 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:17.538432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.527921 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhwj\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-kube-api-access-7fhwj\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.538432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.532193 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-token\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.538432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.538373 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-podinfo\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.541751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.541718 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-envoy\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.544894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.544831 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-credential-socket\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.545245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.545228 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-certs\") pod \"e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.572542 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:32:17.572500 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:32:17.709712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.709638 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:17.925755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:17.925727 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 20:32:18.002422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:18.002396 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc"] Nov 03 20:32:18.003338 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:32:18.003311 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4153e58d_b2c1_48c5_9175_984b7115edb5.slice/crio-09432f0f0a6134878792b38026fca68fe004e57315f8141d086d30404d9b06a6 WatchSource:0}: Error finding container 09432f0f0a6134878792b38026fca68fe004e57315f8141d086d30404d9b06a6: Status 404 returned error can't find the container with id 09432f0f0a6134878792b38026fca68fe004e57315f8141d086d30404d9b06a6 Nov 03 20:32:18.006601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:18.006566 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:32:18.006691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:18.006648 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:32:18.006728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:18.006687 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:32:18.257990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:18.257950 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:19.015373 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:19.015333 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" event={"ID":"4153e58d-b2c1-48c5-9175-984b7115edb5","Type":"ContainerStarted","Data":"e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0"} Nov 03 20:32:19.015591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:19.015390 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" event={"ID":"4153e58d-b2c1-48c5-9175-984b7115edb5","Type":"ContainerStarted","Data":"09432f0f0a6134878792b38026fca68fe004e57315f8141d086d30404d9b06a6"} Nov 03 20:32:19.043614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:19.043562 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podStartSLOduration=2.043549085 podStartE2EDuration="2.043549085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:19.040477372 +0000 UTC m=+1842.490583418" watchObservedRunningTime="2026-11-03 20:32:19.043549085 +0000 UTC m=+1842.493655127" Nov 03 20:32:19.113098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:19.113065 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:19.710116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:19.710080 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:19.715369 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:19.715346 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:19.905205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:19.901998 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:20.022704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:20.022674 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:20.023733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:20.023712 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:32:20.612315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:20.612283 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:21.010647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.010610 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd"] Nov 03 20:32:21.011058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.010843 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" containerID="cri-o://f6f375ca6b2a4c221f991dc71bd75e6a985c782de1bc61f732df094b42808a58" gracePeriod=30 Nov 03 20:32:21.043792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.043759 4711 generic.go:356] "Generic (PLEG): container finished" podID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerID="18305c90fd035110ad9f958b68ff2a84cd22753138d857681f773462927aad03" exitCode=0 Nov 03 20:32:21.043961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.043815 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" event={"ID":"6b598ce1-a56b-42c6-b34e-4054d7fc290c","Type":"ContainerDied","Data":"18305c90fd035110ad9f958b68ff2a84cd22753138d857681f773462927aad03"} Nov 03 20:32:21.152800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.152759 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:21.177934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.177905 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:32:21.300069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.300033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:21.335733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.335702 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-kube-api-access-zrtmd\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-kube-api-access-zrtmd\") pod \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " Nov 03 20:32:21.335938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.335747 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-data\") pod \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " Nov 03 20:32:21.335938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.335804 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-token\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-token\") pod \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " Nov 03 20:32:21.335938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.335859 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-podinfo\") pod \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " Nov 03 20:32:21.335938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.335885 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-credential-socket\") pod \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " Nov 03 20:32:21.336163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.335990 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-socket\") pod \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " Nov 03 20:32:21.336163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.336045 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-envoy\") pod \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " Nov 03 20:32:21.336163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.336106 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-data" pod "6b598ce1-a56b-42c6-b34e-4054d7fc290c" (UID: "6b598ce1-a56b-42c6-b34e-4054d7fc290c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.336282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.336245 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-certs\") pod \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " Nov 03 20:32:21.336282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.336249 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-socket" pod "6b598ce1-a56b-42c6-b34e-4054d7fc290c" (UID: "6b598ce1-a56b-42c6-b34e-4054d7fc290c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.336282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.336259 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-credential-socket" pod "6b598ce1-a56b-42c6-b34e-4054d7fc290c" (UID: "6b598ce1-a56b-42c6-b34e-4054d7fc290c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.336502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.336289 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istiod-ca-cert\") pod \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\" (UID: \"6b598ce1-a56b-42c6-b34e-4054d7fc290c\") " Nov 03 20:32:21.336606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.336553 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-certs" pod "6b598ce1-a56b-42c6-b34e-4054d7fc290c" (UID: "6b598ce1-a56b-42c6-b34e-4054d7fc290c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.336671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.336649 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istiod-ca-cert" pod "6b598ce1-a56b-42c6-b34e-4054d7fc290c" (UID: "6b598ce1-a56b-42c6-b34e-4054d7fc290c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:21.337228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.337202 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.337333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.337232 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.337333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.337248 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.337333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.337261 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.337333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.337277 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.338309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.338288 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-podinfo" pod "6b598ce1-a56b-42c6-b34e-4054d7fc290c" (UID: "6b598ce1-a56b-42c6-b34e-4054d7fc290c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:21.338407 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.338315 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-token" pod "6b598ce1-a56b-42c6-b34e-4054d7fc290c" (UID: "6b598ce1-a56b-42c6-b34e-4054d7fc290c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:21.338490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.338404 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-envoy" pod "6b598ce1-a56b-42c6-b34e-4054d7fc290c" (UID: "6b598ce1-a56b-42c6-b34e-4054d7fc290c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:21.339131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.339111 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-kube-api-access-zrtmd" pod "6b598ce1-a56b-42c6-b34e-4054d7fc290c" (UID: "6b598ce1-a56b-42c6-b34e-4054d7fc290c"). InnerVolumeSpecName "kube-api-access-zrtmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:21.439053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.439023 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrtmd\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-kube-api-access-zrtmd\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.439053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.439049 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.439053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.439059 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.439284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.439068 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b598ce1-a56b-42c6-b34e-4054d7fc290c-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:21.529003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.528965 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv"] Nov 03 20:32:21.529809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.529789 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" Nov 03 20:32:21.529809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.529809 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" Nov 03 20:32:21.530018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.530004 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" containerName="istio-proxy" Nov 03 20:32:21.535553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.535529 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.546770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.546741 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0f6e09-serving-cert\"" Nov 03 20:32:21.547269 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.547243 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv"] Nov 03 20:32:21.599600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.599516 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:21.681470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.671511 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39afa40-37bc-43fc-ade1-a39736d8104f-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.681470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.671609 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-metrics-certs\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.681470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.671832 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.681470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.671948 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgnsk\" (UniqueName: \"kubernetes.io/projected/b39afa40-37bc-43fc-ade1-a39736d8104f-kube-api-access-mgnsk\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.738597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.738490 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:21.773913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.773875 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgnsk\" (UniqueName: \"kubernetes.io/projected/b39afa40-37bc-43fc-ade1-a39736d8104f-kube-api-access-mgnsk\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.774215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.774073 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39afa40-37bc-43fc-ade1-a39736d8104f-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.774215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.774170 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-metrics-certs\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.774362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.774317 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.774719 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:32:21.774695 4711 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-serving-cert: secret "maas-api-e2e-explicit-0f6e09-serving-cert" not found Nov 03 20:32:21.775001 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:32:21.774969 4711 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls podName:b39afa40-37bc-43fc-ade1-a39736d8104f nodeName:}" failed. No retries permitted until 2026-11-03 20:32:22.274948429 +0000 UTC m=+1845.725054453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls") pod "maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" (UID: "b39afa40-37bc-43fc-ade1-a39736d8104f") : secret "maas-api-e2e-explicit-0f6e09-serving-cert" not found Nov 03 20:32:21.775177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.775148 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39afa40-37bc-43fc-ade1-a39736d8104f-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.798600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.798565 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-metrics-certs\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.798757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.798639 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgnsk\" (UniqueName: \"kubernetes.io/projected/b39afa40-37bc-43fc-ade1-a39736d8104f-kube-api-access-mgnsk\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:21.934075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:21.933980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:22.052942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.052913 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" Nov 03 20:32:22.053370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.052920 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9" event={"ID":"6b598ce1-a56b-42c6-b34e-4054d7fc290c","Type":"ContainerDied","Data":"9f2e0c0814c736083fea5067cc496e40254943dc66f137a83563c7f67f8942b0"} Nov 03 20:32:22.053370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.053049 4711 scope.go:122] "RemoveContainer" containerID="18305c90fd035110ad9f958b68ff2a84cd22753138d857681f773462927aad03" Nov 03 20:32:22.101731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.101706 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9"] Nov 03 20:32:22.119369 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.119343 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-qfsz9"] Nov 03 20:32:22.284172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.284134 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:22.286825 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.286798 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls\") pod \"maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:22.384529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.384468 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:22.458268 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.458232 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:22.525102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.524959 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:22.792528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.784843 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:22.855517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.847425 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b598ce1-a56b-42c6-b34e-4054d7fc290c" path="/var/lib/kubelet/pods/6b598ce1-a56b-42c6-b34e-4054d7fc290c/volumes" Nov 03 20:32:22.888511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:22.888416 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv"] Nov 03 20:32:22.888891 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:32:22.888866 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39afa40_37bc_43fc_ade1_a39736d8104f.slice/crio-83da63143397d9de30bddf5281ac7c9b7a807e98e26a9b7d51535f003ea39f54 WatchSource:0}: Error finding container 83da63143397d9de30bddf5281ac7c9b7a807e98e26a9b7d51535f003ea39f54: Status 404 returned error can't find the container with id 83da63143397d9de30bddf5281ac7c9b7a807e98e26a9b7d51535f003ea39f54 Nov 03 20:32:23.071205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:23.071115 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" event={"ID":"b39afa40-37bc-43fc-ade1-a39736d8104f","Type":"ContainerStarted","Data":"83da63143397d9de30bddf5281ac7c9b7a807e98e26a9b7d51535f003ea39f54"} Nov 03 20:32:23.573410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:23.573358 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:23.734524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:23.726594 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:23.865273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:23.865241 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:24.061674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:24.061637 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:24.096717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:24.096681 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" event={"ID":"b39afa40-37bc-43fc-ade1-a39736d8104f","Type":"ContainerStarted","Data":"c8af36246b90c0b1f5b498b00812fa41a33d97fde6b3ed4e95c6d2490e9a166a"} Nov 03 20:32:24.097572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:24.097536 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:24.195992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:24.195958 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:24.397383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:24.397299 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:24.530704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:24.530668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:25.102528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:25.089488 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:25.881928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:25.881895 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:26.169821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.169728 4711 generic.go:356] "Generic (PLEG): container finished" podID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerID="f6f375ca6b2a4c221f991dc71bd75e6a985c782de1bc61f732df094b42808a58" exitCode=0 Nov 03 20:32:26.170180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.169820 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" event={"ID":"2e364d3d-f437-4bd6-b0c0-8963f3cc065f","Type":"ContainerDied","Data":"f6f375ca6b2a4c221f991dc71bd75e6a985c782de1bc61f732df094b42808a58"} Nov 03 20:32:26.333379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.333352 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:32:26.361340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.361260 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" podStartSLOduration=4.821265249 podStartE2EDuration="5.361244712s" totalImagesPullingTime="539.979463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:24.143898735 +0000 UTC m=+1847.594004783" watchObservedRunningTime="2026-11-03 20:32:26.361244712 +0000 UTC m=+1849.811350757" Nov 03 20:32:26.378543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.378512 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-envoy\") pod \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " Nov 03 20:32:26.378723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.378623 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istiod-ca-cert\") pod \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " Nov 03 20:32:26.378723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.378709 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-certs\") pod \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " Nov 03 20:32:26.378854 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.378838 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-data\") pod \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " Nov 03 20:32:26.378911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.378871 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-podinfo\") pod \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " Nov 03 20:32:26.378961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.378923 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-kube-api-access-4t9sm\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-kube-api-access-4t9sm\") pod \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " Nov 03 20:32:26.379019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.378964 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-credential-socket\") pod \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " Nov 03 20:32:26.379070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.379019 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-socket\") pod \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " Nov 03 20:32:26.379070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.379036 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istiod-ca-cert" pod "2e364d3d-f437-4bd6-b0c0-8963f3cc065f" (UID: "2e364d3d-f437-4bd6-b0c0-8963f3cc065f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:26.379178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.379159 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-token\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-token\") pod \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\" (UID: \"2e364d3d-f437-4bd6-b0c0-8963f3cc065f\") " Nov 03 20:32:26.379288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.379270 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-socket" pod "2e364d3d-f437-4bd6-b0c0-8963f3cc065f" (UID: "2e364d3d-f437-4bd6-b0c0-8963f3cc065f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:26.380155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.380130 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.380155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.380156 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.381201 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.381154 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-data" pod "2e364d3d-f437-4bd6-b0c0-8963f3cc065f" (UID: "2e364d3d-f437-4bd6-b0c0-8963f3cc065f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:26.381402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.381371 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-envoy" pod "2e364d3d-f437-4bd6-b0c0-8963f3cc065f" (UID: "2e364d3d-f437-4bd6-b0c0-8963f3cc065f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:26.381561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.381479 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-podinfo" pod "2e364d3d-f437-4bd6-b0c0-8963f3cc065f" (UID: "2e364d3d-f437-4bd6-b0c0-8963f3cc065f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:26.381642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.381583 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-credential-socket" pod "2e364d3d-f437-4bd6-b0c0-8963f3cc065f" (UID: "2e364d3d-f437-4bd6-b0c0-8963f3cc065f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:26.381701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.379046 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-certs" pod "2e364d3d-f437-4bd6-b0c0-8963f3cc065f" (UID: "2e364d3d-f437-4bd6-b0c0-8963f3cc065f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:26.382510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.382436 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-token" pod "2e364d3d-f437-4bd6-b0c0-8963f3cc065f" (UID: "2e364d3d-f437-4bd6-b0c0-8963f3cc065f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:26.382609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.382544 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-kube-api-access-4t9sm" pod "2e364d3d-f437-4bd6-b0c0-8963f3cc065f" (UID: "2e364d3d-f437-4bd6-b0c0-8963f3cc065f"). InnerVolumeSpecName "kube-api-access-4t9sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:26.482214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.482130 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.482214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.482169 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.482214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.482184 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.482214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.482196 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.482214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.482207 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9sm\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-kube-api-access-4t9sm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.482214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.482216 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.482587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.482224 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e364d3d-f437-4bd6-b0c0-8963f3cc065f-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:26.700830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:26.700797 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:27.178611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:27.178579 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" Nov 03 20:32:27.178611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:27.178606 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd" event={"ID":"2e364d3d-f437-4bd6-b0c0-8963f3cc065f","Type":"ContainerDied","Data":"08d6a036bbab77bfc90016a5cd49712817a8ccd5e481cd8b37505730b10e8eea"} Nov 03 20:32:27.179081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:27.178675 4711 scope.go:122] "RemoveContainer" containerID="f6f375ca6b2a4c221f991dc71bd75e6a985c782de1bc61f732df094b42808a58" Nov 03 20:32:27.202893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:27.202862 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd"] Nov 03 20:32:27.207012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:27.206985 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-qstfd"] Nov 03 20:32:27.391033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:27.390997 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:28.472185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:28.472147 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:28.682690 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:28.682658 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" path="/var/lib/kubelet/pods/2e364d3d-f437-4bd6-b0c0-8963f3cc065f/volumes" Nov 03 20:32:29.225987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:29.225956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:29.356212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:29.356175 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:30.155529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.155488 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:32:30.164136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.164104 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn"] Nov 03 20:32:30.164931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.164909 4711 cpu_manager.go:414] "containerMap: removing container" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" Nov 03 20:32:30.164931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.164929 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" Nov 03 20:32:30.165176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.165160 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e364d3d-f437-4bd6-b0c0-8963f3cc065f" containerName="istio-proxy" Nov 03 20:32:30.169251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.169232 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.176197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.176178 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ba89e0-openshift-default-dockercfg-rvpjm\"" Nov 03 20:32:30.182033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.182011 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn"] Nov 03 20:32:30.245585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.241815 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:30.283003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.282962 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.283157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.283071 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-envoy\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.283157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.283103 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-podinfo\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.283316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.283298 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw54q\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-kube-api-access-sw54q\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.283352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.283334 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-credential-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.283423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.283407 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-data\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.283569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.283551 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-certs\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.283618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.283604 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-token\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.283660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.283647 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istiod-ca-cert\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.387886 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw54q\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-kube-api-access-sw54q\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.387945 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-credential-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.388004 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-data\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.388091 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-certs\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.388184 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-token\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.388227 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istiod-ca-cert\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.388339 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.388422 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-envoy\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.388467 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-podinfo\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.389056 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-certs\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.389435 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-data\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.391522 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istiod-ca-cert\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.391754 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.392518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.392227 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-credential-socket\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.393661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.393635 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-envoy\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.397944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.397920 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-podinfo\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.403669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.403647 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-token\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.404132 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.404115 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw54q\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-kube-api-access-sw54q\") pod \"e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.446883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.446793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:30.487823 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.487789 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:30.794168 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:32:30.794127 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2a83b1_1b4a_4624_b0c6_bdb4a275d7db.slice/crio-edd00a89c9f74a359df44b83ac5f88aa573937b11443d5db370edbc8b50f7c8b WatchSource:0}: Error finding container edd00a89c9f74a359df44b83ac5f88aa573937b11443d5db370edbc8b50f7c8b: Status 404 returned error can't find the container with id edd00a89c9f74a359df44b83ac5f88aa573937b11443d5db370edbc8b50f7c8b Nov 03 20:32:30.794168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.794147 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn"] Nov 03 20:32:30.796948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.796915 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:32:30.797046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.796982 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:32:30.797046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:30.797030 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:32:31.206333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:31.206240 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:31.283751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:31.283695 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" event={"ID":"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db","Type":"ContainerStarted","Data":"d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e"} Nov 03 20:32:31.283934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:31.283780 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" event={"ID":"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db","Type":"ContainerStarted","Data":"edd00a89c9f74a359df44b83ac5f88aa573937b11443d5db370edbc8b50f7c8b"} Nov 03 20:32:31.315086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:31.315032 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" podStartSLOduration=1.315016856 podStartE2EDuration="1.315016856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:31.310318591 +0000 UTC m=+1854.760424635" watchObservedRunningTime="2026-11-03 20:32:31.315016856 +0000 UTC m=+1854.765122901" Nov 03 20:32:31.490704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:31.490613 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:31.492754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:31.492712 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:32:31.492857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:31.492784 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 20:32:31.820949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:31.820901 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:31.996976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:31.996943 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:32.304031 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:32.303992 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:32.456468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:32.453940 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:32.505468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:32.491782 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:32:32.505468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:32.491840 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 20:32:33.147210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:33.147176 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:33.490898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:33.490804 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:32:33.490898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:33.490874 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 20:32:34.045500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:34.045414 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:34.490527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:34.489822 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:32:34.490527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:34.489885 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 20:32:34.822039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:34.821999 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:35.490869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:35.490831 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:32:35.491080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:35.490906 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 20:32:35.629228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:35.629193 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:35.892568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:35.892528 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:36.046377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:36.046292 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:36.490862 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:36.490825 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 20:32:36.491037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:36.490902 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 20:32:36.766059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:36.766028 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:36.904818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:36.904777 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:37.151847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:37.151751 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:37.293046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:37.292997 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:37.511786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:37.511758 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:37.513542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:37.513520 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:37.534841 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:37.534810 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:32:38.097232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:38.097196 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:38.874062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:38.874032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:39.457185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.457152 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz"] Nov 03 20:32:39.463216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.463195 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.469846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.469824 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-6e5139-openshift-default-dockercfg-7m2n7\"" Nov 03 20:32:39.481263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.481239 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz"] Nov 03 20:32:39.516276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.516246 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-envoy\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.516429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.516353 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-credential-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.516528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.516423 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgkdj\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-kube-api-access-bgkdj\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.516528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.516511 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996362b4-6f32-4168-b8de-2433bf0f5cf4-istiod-ca-cert\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.516617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.516574 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-token\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.516678 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.516619 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-data\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.516742 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.516685 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-podinfo\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.517024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.516992 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-certs\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.517133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.517107 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.635881 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-envoy\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.635982 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-credential-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.636065 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkdj\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-kube-api-access-bgkdj\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.636103 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996362b4-6f32-4168-b8de-2433bf0f5cf4-istiod-ca-cert\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.636197 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-token\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.636248 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-data\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.636274 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-podinfo\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.636342 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-certs\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.637227 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996362b4-6f32-4168-b8de-2433bf0f5cf4-istiod-ca-cert\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.641253 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.646142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.641848 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-credential-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.678508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.664580 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-certs\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.678508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.665019 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-data\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.709376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.707334 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-podinfo\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.720614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.711478 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-socket\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.720788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.720756 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkdj\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-kube-api-access-bgkdj\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.739501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.730052 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-envoy\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.739501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.730897 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-token\") pod \"e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.779469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.778231 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:39.940253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:39.940197 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:40.095081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.095034 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz"] Nov 03 20:32:40.100809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.100773 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:32:40.100906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.100842 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:32:40.100906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.100872 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:32:40.432324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.432227 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" event={"ID":"996362b4-6f32-4168-b8de-2433bf0f5cf4","Type":"ContainerStarted","Data":"231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e"} Nov 03 20:32:40.432324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.432281 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" event={"ID":"996362b4-6f32-4168-b8de-2433bf0f5cf4","Type":"ContainerStarted","Data":"91d9cfa1341fc495ff376fc6a2d5c11abd26082ab0ae6aa8a47ddce9acc12fed"} Nov 03 20:32:40.458968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.458914 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" podStartSLOduration=1.458900611 podStartE2EDuration="1.458900611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:40.456371824 +0000 UTC m=+1863.906477870" watchObservedRunningTime="2026-11-03 20:32:40.458900611 +0000 UTC m=+1863.909006655" Nov 03 20:32:40.785468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.779539 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:40.829899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.829859 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:40.965290 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:40.965257 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:41.812471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:41.793907 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:32:41.812471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:41.793989 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:32:41.812471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:41.794657 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:42.237080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:42.237001 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:42.789477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:42.786226 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:32:42.789477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:42.786289 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:32:42.962014 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:42.961964 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:43.199058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:43.198935 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:43.789549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:43.788920 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:32:43.789549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:43.788988 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:32:44.110723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:44.110637 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:44.783857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:44.783817 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:32:44.784147 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:44.784119 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:32:44.800179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:44.800145 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:45.603979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:45.603945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:45.779518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:45.779481 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:32:45.779795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:45.779767 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:32:46.428285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:46.428242 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:46.656934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:46.656900 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:46.784312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:46.784279 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:46.784770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:46.784754 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:46.785537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:46.785518 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 20:32:47.272023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:47.271983 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:47.534036 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:32:47.533959 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:32:47.993069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:47.992991 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:48.718300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:48.718264 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:49.421430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:49.421375 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:50.153875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:50.153839 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.045178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:51.045138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.188804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:51.188769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:51.622311 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:51.622280 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:52.212468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:52.211050 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:52.974363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:52.974319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:53.116930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:53.116894 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:54.033543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:54.033507 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:55.417134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:55.417089 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.695380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:56.695347 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:56.847500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:56.847468 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:57.493005 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:57.492971 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:57.639806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:57.639763 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.092850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:58.092803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.225307 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:58.225251 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.447533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:58.447437 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:58.600415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:58.600371 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:59.153063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:59.153026 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:32:59.592025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:32:59.591988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:00.489469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:00.489372 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:01.585003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:01.584968 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:02.327248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:02.327214 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:02.462813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:02.462781 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:02.636315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:02.636216 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:03.479598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.479568 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv"] Nov 03 20:33:03.485077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.485038 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.497529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.497500 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv"] Nov 03 20:33:03.587734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.587702 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-9fbc61-openshift-default-dockercfg-v2stb\"" Nov 03 20:33:03.593972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.593939 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.594135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.593999 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.594190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.594154 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.594319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.594287 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.594499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.594439 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.594594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.594574 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247br\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-kube-api-access-247br\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.594794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.594772 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.594893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.594813 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.594893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.594854 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.633675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.633639 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:03.696440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696406 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.696953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696462 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.696953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696504 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.696953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696540 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.696953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696578 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.696953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696607 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.696953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696676 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.696953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696790 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.696953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696883 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247br\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-kube-api-access-247br\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.697374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.696966 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-credential-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.697374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.697162 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-data\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.697374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.697182 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-socket\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.697374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.697346 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-certs\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.697613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.697479 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istiod-ca-cert\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.698925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.698899 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-envoy\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.699415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.699391 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-podinfo\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.709637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.709605 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-token\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.710242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.710219 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-247br\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-kube-api-access-247br\") pod \"e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:03.776795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.776721 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:03.859468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:03.856416 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:04.383977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:04.383948 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv"] Nov 03 20:33:04.384678 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:33:04.384645 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b65a96d_1374_4b61_b3ed_3a12eb395abf.slice/crio-361199e453329cd523078bd3a12b403a8679ecb19e4ddc0dee6e0400c164554d WatchSource:0}: Error finding container 361199e453329cd523078bd3a12b403a8679ecb19e4ddc0dee6e0400c164554d: Status 404 returned error can't find the container with id 361199e453329cd523078bd3a12b403a8679ecb19e4ddc0dee6e0400c164554d Nov 03 20:33:04.387883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:04.387848 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:33:04.387987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:04.387930 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:33:04.388056 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:04.387998 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:33:04.692999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:04.692904 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:05.069601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:05.069560 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" event={"ID":"5b65a96d-1374-4b61-b3ed-3a12eb395abf","Type":"ContainerStarted","Data":"6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5"} Nov 03 20:33:05.069999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:05.069625 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" event={"ID":"5b65a96d-1374-4b61-b3ed-3a12eb395abf","Type":"ContainerStarted","Data":"361199e453329cd523078bd3a12b403a8679ecb19e4ddc0dee6e0400c164554d"} Nov 03 20:33:05.098701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:05.098656 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" podStartSLOduration=2.09864206 podStartE2EDuration="2.09864206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:05.096433161 +0000 UTC m=+1888.546539211" watchObservedRunningTime="2026-11-03 20:33:05.09864206 +0000 UTC m=+1888.548748105" Nov 03 20:33:05.493784 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:05.493416 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:05.990471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:05.989841 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:05.992572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:05.991913 4711 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 20:33:05.992572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:05.991988 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 20:33:06.064593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:06.064557 4711 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:33:06.064769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:06.064607 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:06.305707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:06.301036 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:06.590568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:06.588398 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:06.984522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:06.978534 4711 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 20:33:06.984522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:06.978604 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 20:33:07.110302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:07.110269 4711 generic.go:356] "Generic (PLEG): container finished" podID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerID="e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0" exitCode=0 Nov 03 20:33:07.110490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:07.110341 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" event={"ID":"4153e58d-b2c1-48c5-9175-984b7115edb5","Type":"ContainerDied","Data":"e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0"} Nov 03 20:33:07.110859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:07.110844 4711 scope.go:122] "RemoveContainer" containerID="e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0" Nov 03 20:33:07.112729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:07.112702 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:33:07.112838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:07.112788 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:33:07.112838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:07.112821 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:33:07.163057 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:07.163026 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:07.973476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:07.972236 4711 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 20:33:07.973476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:07.972289 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 20:33:08.036288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.036200 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:08.152503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.152323 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" event={"ID":"4153e58d-b2c1-48c5-9175-984b7115edb5","Type":"ContainerStarted","Data":"412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8"} Nov 03 20:33:08.180555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.180522 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:08.561804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.561765 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:08.809114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.792319 4711 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:08.809114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.792373 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 20:33:08.904588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.888988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:08.904588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.893657 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:33:08.970848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.968918 4711 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 20:33:08.970848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:08.968994 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 20:33:09.130278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:09.128894 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:09.714468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:09.713869 4711 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:09.714468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:09.713931 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 20:33:09.940439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:09.940399 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:09.981011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:09.980902 4711 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 20:33:09.982589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:09.982548 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 20:33:10.513475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:10.510477 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:10.714517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:10.714479 4711 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:10.714809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:10.714780 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 20:33:10.943465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:10.943329 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:10.988468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:10.988005 4711 patch_prober.go:29] interesting pod/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 20:33:10.988468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:10.988082 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 20:33:11.104471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:11.103242 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:11.495909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:11.492379 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:11.710011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:11.709963 4711 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:11.710482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:11.710022 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 20:33:11.990914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:11.990879 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:11.992574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:11.992553 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:11.993415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:11.993393 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:33:12.119665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:12.119633 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:12.687470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:12.684085 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:12.712508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:12.711008 4711 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:12.712508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:12.711067 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 20:33:12.821133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:12.821094 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:13.029004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:13.028971 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:13.295005 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:13.294909 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:13.519318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:13.510402 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:13.679650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:13.679564 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:13.710950 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:13.710920 4711 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 03 20:33:13.711118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:13.710981 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 03 20:33:14.060012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:14.059977 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.207502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:14.207005 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.715122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:14.715089 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:33:14.715994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:14.715962 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:33:14.717233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:14.717214 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 20:33:14.828146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:14.828081 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:14.961050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:14.961012 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:15.958480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:15.958414 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:16.107610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:16.107572 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:16.241064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:16.240984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:17.338511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:17.328769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:17.534493 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:33:17.534435 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:33:18.018805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:18.018764 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:18.812219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:18.812168 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:19.501047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:19.501010 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.184324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:20.184289 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.328757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:20.328374 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:20.464144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:20.464046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:21.153019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:21.152983 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.003001 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:22.002953 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:22.701278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:22.701237 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:23.420106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:23.420068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:24.108061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:24.108028 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:24.796462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:24.796409 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:25.463600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:25.463566 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:26.219465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:26.219415 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:26.917308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:26.917273 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:27.616218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:27.613356 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:28.282361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:28.282320 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:29.031213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:29.031182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:29.884472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:29.883218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:30.710341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:30.710305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:30.841279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:30.841245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:30.994557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:30.994467 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:32.109437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:32.109404 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:32.958820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:32.958779 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:33.317638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:33.317598 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:33.452982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:33.452946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.196199 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:34.196163 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.621792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:34.621749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:34.921998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:34.921903 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:35.625437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:35.625396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:36.304392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:36.304348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:36.960118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:36.960084 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:37.267294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:37.267253 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:37.950909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:37.950871 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:38.803049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:38.803009 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:39.498733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:39.498697 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.163036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:40.163000 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.801149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:40.801119 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:40.928552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:40.928514 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:41.518584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:41.518543 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.201192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:42.201152 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.860028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:42.859989 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:42.987481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:42.987429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:43.712560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:43.712522 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:44.177561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:44.177499 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:44.310568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:44.310539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:44.471131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:44.471043 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:44.597123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:44.597090 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:44.763251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:44.763221 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:44.913914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:44.913878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:45.050749 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:45.050660 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:45.331021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:45.330921 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:45.504518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:45.504440 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:45.634647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:45.634561 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:46.199247 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:46.199213 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:46.620467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:46.619076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:47.455299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:47.452320 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:47.541593 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:33:47.541534 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:33:48.176072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:48.175984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:49.209019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:49.208984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:49.736720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:49.736679 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv"] Nov 03 20:33:49.737270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:49.737232 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" podUID="b39afa40-37bc-43fc-ade1-a39736d8104f" containerName="maas-api" containerID="cri-o://c8af36246b90c0b1f5b498b00812fa41a33d97fde6b3ed4e95c6d2490e9a166a" gracePeriod=30 Nov 03 20:33:49.819353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:49.819314 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:49.918576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:49.918546 4711 generic.go:356] "Generic (PLEG): container finished" podID="b39afa40-37bc-43fc-ade1-a39736d8104f" containerID="c8af36246b90c0b1f5b498b00812fa41a33d97fde6b3ed4e95c6d2490e9a166a" exitCode=0 Nov 03 20:33:49.918704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:49.918672 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" event={"ID":"b39afa40-37bc-43fc-ade1-a39736d8104f","Type":"ContainerDied","Data":"c8af36246b90c0b1f5b498b00812fa41a33d97fde6b3ed4e95c6d2490e9a166a"} Nov 03 20:33:50.129469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.126180 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:50.342423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.342394 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:33:50.375546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.375516 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39afa40-37bc-43fc-ade1-a39736d8104f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39afa40-37bc-43fc-ade1-a39736d8104f-openshift-service-ca\") pod \"b39afa40-37bc-43fc-ade1-a39736d8104f\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " Nov 03 20:33:50.375734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.375685 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls\") pod \"b39afa40-37bc-43fc-ade1-a39736d8104f\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " Nov 03 20:33:50.375734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.375720 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-metrics-certs\") pod \"b39afa40-37bc-43fc-ade1-a39736d8104f\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " Nov 03 20:33:50.375926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.375901 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39afa40-37bc-43fc-ade1-a39736d8104f-kube-api-access-mgnsk\" (UniqueName: \"kubernetes.io/projected/b39afa40-37bc-43fc-ade1-a39736d8104f-kube-api-access-mgnsk\") pod \"b39afa40-37bc-43fc-ade1-a39736d8104f\" (UID: \"b39afa40-37bc-43fc-ade1-a39736d8104f\") " Nov 03 20:33:50.376148 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.375922 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39afa40-37bc-43fc-ade1-a39736d8104f-openshift-service-ca" pod "b39afa40-37bc-43fc-ade1-a39736d8104f" (UID: "b39afa40-37bc-43fc-ade1-a39736d8104f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:50.377089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.377059 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b39afa40-37bc-43fc-ade1-a39736d8104f-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:50.378262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.378236 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39afa40-37bc-43fc-ade1-a39736d8104f-kube-api-access-mgnsk" pod "b39afa40-37bc-43fc-ade1-a39736d8104f" (UID: "b39afa40-37bc-43fc-ade1-a39736d8104f"). InnerVolumeSpecName "kube-api-access-mgnsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:50.378333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.378294 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-metrics-certs" pod "b39afa40-37bc-43fc-ade1-a39736d8104f" (UID: "b39afa40-37bc-43fc-ade1-a39736d8104f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:50.378435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.378417 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls" pod "b39afa40-37bc-43fc-ade1-a39736d8104f" (UID: "b39afa40-37bc-43fc-ade1-a39736d8104f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:50.396667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.396631 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:50.495977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.495937 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:50.501052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.501018 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b39afa40-37bc-43fc-ade1-a39736d8104f-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:50.501318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.501285 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgnsk\" (UniqueName: \"kubernetes.io/projected/b39afa40-37bc-43fc-ade1-a39736d8104f-kube-api-access-mgnsk\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:50.926947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.926907 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" event={"ID":"b39afa40-37bc-43fc-ade1-a39736d8104f","Type":"ContainerDied","Data":"83da63143397d9de30bddf5281ac7c9b7a807e98e26a9b7d51535f003ea39f54"} Nov 03 20:33:50.926947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.926947 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" Nov 03 20:33:50.927177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.926969 4711 scope.go:122] "RemoveContainer" containerID="c8af36246b90c0b1f5b498b00812fa41a33d97fde6b3ed4e95c6d2490e9a166a" Nov 03 20:33:50.949553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.949477 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv"] Nov 03 20:33:50.953722 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:50.953695 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv"] Nov 03 20:33:51.093265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.093230 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:51.435819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.435785 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:51.803082 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.803050 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp"] Nov 03 20:33:51.803805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.803789 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b39afa40-37bc-43fc-ade1-a39736d8104f" containerName="maas-api" Nov 03 20:33:51.803890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.803806 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39afa40-37bc-43fc-ade1-a39736d8104f" containerName="maas-api" Nov 03 20:33:51.803983 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.803970 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39afa40-37bc-43fc-ade1-a39736d8104f" containerName="maas-api" Nov 03 20:33:51.809011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.808983 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:51.813882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.813858 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-03c633-openshift-default-dockercfg-4k2z2\"" Nov 03 20:33:51.820707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.820685 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp"] Nov 03 20:33:51.844831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.844796 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:51.948202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.941802 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68zlj\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-kube-api-access-68zlj\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:51.960611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:51.960569 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-certs\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.001650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.001602 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-data\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.002015 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.001993 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.002577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.002553 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774c317c-a849-4a4f-9921-18004a104b10-istiod-ca-cert\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.002787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.002755 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-credential-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.003055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.003025 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774c317c-a849-4a4f-9921-18004a104b10-istio-podinfo\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.003242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.003212 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-envoy\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.003469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.003427 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-istio-token\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.060517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.044690 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:52.141166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.123232 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68zlj\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-kube-api-access-68zlj\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.143806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.143774 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-certs\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.144041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.144020 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-data\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.157750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.144149 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-certs\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.157750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.144223 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.157750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.144385 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-data\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.157750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.144634 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.162039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.158864 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774c317c-a849-4a4f-9921-18004a104b10-istiod-ca-cert\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.162039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.158928 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-credential-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.162039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.159002 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774c317c-a849-4a4f-9921-18004a104b10-istio-podinfo\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.162039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.159032 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-envoy\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.162039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.159075 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-istio-token\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.166787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.165779 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-credential-socket\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.170882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.167323 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774c317c-a849-4a4f-9921-18004a104b10-istiod-ca-cert\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.200301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.194679 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68zlj\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-kube-api-access-68zlj\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.200301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.195251 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-envoy\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.200301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.198013 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774c317c-a849-4a4f-9921-18004a104b10-istio-podinfo\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.202216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.202196 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-istio-token\") pod \"e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.229137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.229049 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:52.355955 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.355873 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:52.426560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.426521 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:52.696982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.694853 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39afa40-37bc-43fc-ade1-a39736d8104f" path="/var/lib/kubelet/pods/b39afa40-37bc-43fc-ade1-a39736d8104f/volumes" Nov 03 20:33:52.931542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.931512 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp"] Nov 03 20:33:52.931872 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:33:52.931850 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774c317c_a849_4a4f_9921_18004a104b10.slice/crio-6af47a22b4348b65dd4ffc0c4dd3d1f97bc1cc98b6d46aaa8053f44bfa804224 WatchSource:0}: Error finding container 6af47a22b4348b65dd4ffc0c4dd3d1f97bc1cc98b6d46aaa8053f44bfa804224: Status 404 returned error can't find the container with id 6af47a22b4348b65dd4ffc0c4dd3d1f97bc1cc98b6d46aaa8053f44bfa804224 Nov 03 20:33:52.934567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.934536 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:33:52.934647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.934603 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:33:52.934647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:52.934634 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:33:53.088503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:53.085794 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" event={"ID":"774c317c-a849-4a4f-9921-18004a104b10","Type":"ContainerStarted","Data":"89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d"} Nov 03 20:33:53.088503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:53.085898 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" event={"ID":"774c317c-a849-4a4f-9921-18004a104b10","Type":"ContainerStarted","Data":"6af47a22b4348b65dd4ffc0c4dd3d1f97bc1cc98b6d46aaa8053f44bfa804224"} Nov 03 20:33:53.114560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:53.111675 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" podStartSLOduration=2.11166172 podStartE2EDuration="2.11166172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:53.108888211 +0000 UTC m=+1936.558994270" watchObservedRunningTime="2026-11-03 20:33:53.11166172 +0000 UTC m=+1936.561767767" Nov 03 20:33:53.152323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:53.152278 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f6e09-6cd59b9fb9-7xtrv" podUID="b39afa40-37bc-43fc-ade1-a39736d8104f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.130:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:53.202319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:53.202218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:53.426893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:53.426843 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:53.898609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:53.898566 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:54.035378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:54.035347 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:54.432890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:54.432862 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:54.800021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:54.799984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:55.101499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:55.101368 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:55.102441 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:55.102419 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:33:55.286821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:55.286786 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:55.451572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:55.451433 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:55.907559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:55.907526 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:56.477999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:56.477963 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:56.613176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:56.613141 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:56.797538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:56.790094 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:56.970498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:56.965356 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:57.314415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:57.313811 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:57.557489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:57.557401 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:57.692641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:57.692556 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:58.292149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:58.292113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:58.445836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:58.443557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:59.326065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:59.326028 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:33:59.574702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:33:59.574668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.143416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:00.143380 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.292254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:00.292221 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.422811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:00.422730 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:00.556497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:00.556459 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:01.207490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:01.206410 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:02.018102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:02.018060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:02.763577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:02.763540 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:03.567238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:03.567197 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:03.698143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:03.698105 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:04.710384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:04.710336 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:04.849945 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:04.849907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:05.633171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:05.633136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:06.606439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:06.606403 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:07.576372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:07.576344 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:08.226808 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:08.226776 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:09.015632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:09.015584 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:09.717760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:09.717725 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:10.305006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:10.304971 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:11.125864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:11.125822 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:11.826515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:11.826479 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:12.575634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:12.575603 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:12.706729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:12.706671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:13.420773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:13.420726 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:14.220339 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:14.220279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:14.928092 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:14.928055 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:15.645477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:15.645407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:16.496104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:16.496041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:16.625236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:16.625202 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:17.008528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:17.008493 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:17.314064 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:17.313975 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv"] Nov 03 20:34:17.314242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:17.314219 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" containerID="cri-o://6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5" gracePeriod=30 Nov 03 20:34:17.391620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:17.391587 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:17.545469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:17.541233 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:17.687708 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:34:17.682694 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:34:18.584646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:18.584595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:19.155669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:19.155636 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:19.438223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:19.438132 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:19.610325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:19.610293 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:19.758527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:19.755031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:20.315523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.315427 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb"] Nov 03 20:34:20.320270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.320248 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.330256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.329595 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-37aa95-serving-cert\"" Nov 03 20:34:20.335861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.335834 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb"] Nov 03 20:34:20.346150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.346104 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a690b0-4534-4550-93dd-313405071b7e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.346318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.346300 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znhbn\" (UniqueName: \"kubernetes.io/projected/60a690b0-4534-4550-93dd-313405071b7e-kube-api-access-znhbn\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.346369 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.346356 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-metrics-certs\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.346409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.346381 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.404533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.404414 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:20.448114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.448078 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-metrics-certs\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.448324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.448123 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.448324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.448299 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a690b0-4534-4550-93dd-313405071b7e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.448504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.448485 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znhbn\" (UniqueName: \"kubernetes.io/projected/60a690b0-4534-4550-93dd-313405071b7e-kube-api-access-znhbn\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.448994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.448965 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a690b0-4534-4550-93dd-313405071b7e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.450732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.450706 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.450819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.450756 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-metrics-certs\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.460685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.460663 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znhbn\" (UniqueName: \"kubernetes.io/projected/60a690b0-4534-4550-93dd-313405071b7e-kube-api-access-znhbn\") pod \"maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:20.541176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.540980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:20.648685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:20.648610 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:21.024951 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:34:21.024297 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a690b0_4534_4550_93dd_313405071b7e.slice/crio-28543e9ba4241b4497443884b7ee3253eb25acc109e618db60e30ba8567c9579 WatchSource:0}: Error finding container 28543e9ba4241b4497443884b7ee3253eb25acc109e618db60e30ba8567c9579: Status 404 returned error can't find the container with id 28543e9ba4241b4497443884b7ee3253eb25acc109e618db60e30ba8567c9579 Nov 03 20:34:21.025917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:21.025888 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb"] Nov 03 20:34:21.029683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:21.029655 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:34:21.304642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:21.304545 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:21.663877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:21.661878 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" event={"ID":"60a690b0-4534-4550-93dd-313405071b7e","Type":"ContainerStarted","Data":"28543e9ba4241b4497443884b7ee3253eb25acc109e618db60e30ba8567c9579"} Nov 03 20:34:22.251888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.251841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:22.610872 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.610847 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:34:22.627349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627319 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-kube-api-access-247br\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-kube-api-access-247br\") pod \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " Nov 03 20:34:22.627522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627362 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istiod-ca-cert\") pod \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " Nov 03 20:34:22.627637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627618 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-data\") pod \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " Nov 03 20:34:22.627801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627784 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-certs\") pod \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " Nov 03 20:34:22.627916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627850 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-socket\") pod \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " Nov 03 20:34:22.627916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627850 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-data" pod "5b65a96d-1374-4b61-b3ed-3a12eb395abf" (UID: "5b65a96d-1374-4b61-b3ed-3a12eb395abf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:22.627916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627887 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istiod-ca-cert" pod "5b65a96d-1374-4b61-b3ed-3a12eb395abf" (UID: "5b65a96d-1374-4b61-b3ed-3a12eb395abf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:22.627916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627893 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-envoy\") pod \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " Nov 03 20:34:22.628102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627934 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-token\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-token\") pod \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " Nov 03 20:34:22.628102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627969 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-podinfo\") pod \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " Nov 03 20:34:22.628102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.627999 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-credential-socket\") pod \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\" (UID: \"5b65a96d-1374-4b61-b3ed-3a12eb395abf\") " Nov 03 20:34:22.628102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.628008 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-certs" pod "5b65a96d-1374-4b61-b3ed-3a12eb395abf" (UID: "5b65a96d-1374-4b61-b3ed-3a12eb395abf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:22.628102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.628098 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-socket" pod "5b65a96d-1374-4b61-b3ed-3a12eb395abf" (UID: "5b65a96d-1374-4b61-b3ed-3a12eb395abf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:22.628370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.628321 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-credential-socket" pod "5b65a96d-1374-4b61-b3ed-3a12eb395abf" (UID: "5b65a96d-1374-4b61-b3ed-3a12eb395abf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:22.628758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.628731 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.628877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.628765 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.628877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.628776 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.628877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.628784 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.628877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.628793 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.630037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.630016 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-kube-api-access-247br" pod "5b65a96d-1374-4b61-b3ed-3a12eb395abf" (UID: "5b65a96d-1374-4b61-b3ed-3a12eb395abf"). InnerVolumeSpecName "kube-api-access-247br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:22.630139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.630053 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-envoy" pod "5b65a96d-1374-4b61-b3ed-3a12eb395abf" (UID: "5b65a96d-1374-4b61-b3ed-3a12eb395abf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:22.630190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.630175 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-podinfo" pod "5b65a96d-1374-4b61-b3ed-3a12eb395abf" (UID: "5b65a96d-1374-4b61-b3ed-3a12eb395abf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:22.630285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.630265 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-token" pod "5b65a96d-1374-4b61-b3ed-3a12eb395abf" (UID: "5b65a96d-1374-4b61-b3ed-3a12eb395abf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:22.674682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.674651 4711 generic.go:356] "Generic (PLEG): container finished" podID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerID="6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5" exitCode=0 Nov 03 20:34:22.675046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.674759 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:34:22.684723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.684699 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:22.684910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.684739 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" event={"ID":"60a690b0-4534-4550-93dd-313405071b7e","Type":"ContainerStarted","Data":"4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99"} Nov 03 20:34:22.684910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.684766 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" event={"ID":"5b65a96d-1374-4b61-b3ed-3a12eb395abf","Type":"ContainerDied","Data":"6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5"} Nov 03 20:34:22.684910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.684788 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" event={"ID":"5b65a96d-1374-4b61-b3ed-3a12eb395abf","Type":"ContainerDied","Data":"361199e453329cd523078bd3a12b403a8679ecb19e4ddc0dee6e0400c164554d"} Nov 03 20:34:22.684910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.684809 4711 scope.go:122] "RemoveContainer" containerID="6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5" Nov 03 20:34:22.701781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.701726 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" podStartSLOduration=2.080984798 podStartE2EDuration="2.701706607s" totalImagesPullingTime="620.721809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:22.695052604 +0000 UTC m=+1966.145158649" watchObservedRunningTime="2026-11-03 20:34:22.701706607 +0000 UTC m=+1966.151812656" Nov 03 20:34:22.714692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.714666 4711 scope.go:122] "RemoveContainer" containerID="6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5" Nov 03 20:34:22.715075 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:34:22.715052 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5\": container with ID starting with 6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5 not found: ID does not exist" containerID="6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5" Nov 03 20:34:22.715154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.715083 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5"} err="failed to get container status \"6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5\": rpc error: code = NotFound desc = could not find container \"6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5\": container with ID starting with 6f2b0eb0f90eaf4510e29614734357154be79dedb851bb12413292f2238a76c5 not found: ID does not exist" Nov 03 20:34:22.730782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.730759 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.730782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.730784 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.730925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.730796 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b65a96d-1374-4b61-b3ed-3a12eb395abf-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:22.730925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:22.730809 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-247br\" (UniqueName: \"kubernetes.io/projected/5b65a96d-1374-4b61-b3ed-3a12eb395abf-kube-api-access-247br\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:23.197098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:23.197026 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:23.344078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:23.344037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:23.484249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:23.484162 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:24.016171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:24.016141 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:24.599729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:24.599693 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:24.742229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:24.742198 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:25.315769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:25.315737 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:26.095945 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:26.095909 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:26.903224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:26.903181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:27.604772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:27.604727 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:28.287522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:28.287438 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:28.689260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:28.689190 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 20:34:29.065543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:29.063066 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:29.816861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:29.816588 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:30.703466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:30.696656 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:31.679378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:31.679336 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:32.694125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:32.694093 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:33.703090 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:33.703027 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:34.692379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:34.692345 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:35.358943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:35.358908 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:35.981495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:35.972434 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:36.143291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:36.143257 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:36.925410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:36.924195 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.208345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:37.208241 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.335924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:37.335882 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.489527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:37.489371 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.792568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:37.792535 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:37.958602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:37.958556 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:38.159309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:38.159222 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:38.527018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:38.526978 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:38.674531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:38.669226 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:38.906564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:38.905514 4711 scope.go:122] "RemoveContainer" containerID="099ebb3797e2b1bc5275ef84a99af50bb4631a6c6f36be47fa52b77790e8381c" Nov 03 20:34:39.240100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:39.240017 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:39.450080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:39.438255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:40.399429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:40.399396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:41.157584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.157544 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7"] Nov 03 20:34:41.158787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.158755 4711 cpu_manager.go:414] "containerMap: removing container" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" Nov 03 20:34:41.158787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.158783 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" Nov 03 20:34:41.159034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.159016 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" containerName="istio-proxy" Nov 03 20:34:41.164405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.164377 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.175606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.171198 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-14ebbb-serving-cert\"" Nov 03 20:34:41.182489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.182460 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7"] Nov 03 20:34:41.256429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.256397 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:41.277157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.277128 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.277321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.277265 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaf40-5996-4050-bc42-c543a85cb8a6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.277321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.277291 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mbzg\" (UniqueName: \"kubernetes.io/projected/fcccaf40-5996-4050-bc42-c543a85cb8a6-kube-api-access-8mbzg\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.277485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.277456 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-metrics-certs\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.380013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.379964 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.380165 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.380119 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaf40-5996-4050-bc42-c543a85cb8a6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.380165 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:34:41.380141 4711 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-serving-cert: secret "maas-api-e2e-extmod-a-14ebbb-serving-cert" not found Nov 03 20:34:41.380297 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:34:41.380239 4711 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls podName:fcccaf40-5996-4050-bc42-c543a85cb8a6 nodeName:}" failed. No retries permitted until 2026-11-03 20:34:41.880215373 +0000 UTC m=+1985.330321420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls") pod "maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" (UID: "fcccaf40-5996-4050-bc42-c543a85cb8a6") : secret "maas-api-e2e-extmod-a-14ebbb-serving-cert" not found Nov 03 20:34:41.380363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.380152 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mbzg\" (UniqueName: \"kubernetes.io/projected/fcccaf40-5996-4050-bc42-c543a85cb8a6-kube-api-access-8mbzg\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.380551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.380519 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-metrics-certs\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.380936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.380912 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaf40-5996-4050-bc42-c543a85cb8a6-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.383753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.383721 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-metrics-certs\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.390614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.390565 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:41.393266 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.393243 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mbzg\" (UniqueName: \"kubernetes.io/projected/fcccaf40-5996-4050-bc42-c543a85cb8a6-kube-api-access-8mbzg\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.557862 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.557822 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:41.890515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.888931 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:41.903497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:41.895217 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls\") pod \"maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:42.104010 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:42.103978 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:42.330275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:42.330224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:42.554465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:42.553605 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7"] Nov 03 20:34:42.969021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:42.968917 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" event={"ID":"fcccaf40-5996-4050-bc42-c543a85cb8a6","Type":"ContainerStarted","Data":"73211205f37a1c5c575a4c0deae0991a81a0ca8a03140da64585ebd257bda8c2"} Nov 03 20:34:43.218501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:43.218385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:43.906439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:43.906397 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:44.002763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:44.002716 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" event={"ID":"fcccaf40-5996-4050-bc42-c543a85cb8a6","Type":"ContainerStarted","Data":"9859cad44ee55f020f95300c948cb83e72c2cf5b325b7c6867b8445aa90ed892"} Nov 03 20:34:44.003272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:44.002835 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:44.028682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:44.028618 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" podStartSLOduration=2.551446076 podStartE2EDuration="3.028601539s" totalImagesPullingTime="477.155463ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:44.023507543 +0000 UTC m=+1987.473613588" watchObservedRunningTime="2026-11-03 20:34:44.028601539 +0000 UTC m=+1987.478707625" Nov 03 20:34:44.263233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:44.263194 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:44.714683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:44.714646 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:45.011360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:45.011329 4711 generic.go:356] "Generic (PLEG): container finished" podID="774c317c-a849-4a4f-9921-18004a104b10" containerID="89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d" exitCode=0 Nov 03 20:34:45.011781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:45.011398 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" event={"ID":"774c317c-a849-4a4f-9921-18004a104b10","Type":"ContainerDied","Data":"89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d"} Nov 03 20:34:45.012062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:45.012048 4711 scope.go:122] "RemoveContainer" containerID="89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d" Nov 03 20:34:45.013737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:45.013708 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:34:45.013840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:45.013785 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:34:45.013840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:45.013820 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:34:45.424767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:45.424672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:45.624638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:45.624607 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:46.020385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:46.020346 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" event={"ID":"774c317c-a849-4a4f-9921-18004a104b10","Type":"ContainerStarted","Data":"094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be"} Nov 03 20:34:46.149485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:46.149430 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:46.431179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:46.430933 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:34:46.431544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:46.431517 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:46.432105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:46.432081 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 20:34:46.637152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:46.637120 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:46.769469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:46.767323 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:47.427124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:47.427092 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:47.427560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:47.427163 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 20:34:47.510242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:47.510182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:47.621022 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:34:47.620987 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:34:48.149059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:48.148981 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:48.427827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:48.427741 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:48.427827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:48.427795 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 20:34:48.892131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:48.892091 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:49.427216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:49.427178 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:49.427399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:49.427234 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 20:34:49.724721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:49.724629 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:50.017509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:50.017482 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 20:34:50.308982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:50.308880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:50.427329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:50.427295 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:50.427533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:50.427376 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 20:34:50.435949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:50.435890 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:50.571527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:50.570365 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:51.278659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:51.278620 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:51.427593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:51.427553 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 03 20:34:51.427768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:51.427633 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 03 20:34:51.551904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:51.551807 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:51.717326 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:51.717294 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:52.351759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:52.351721 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:52.433267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:52.433238 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:34:52.433726 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:52.433709 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:34:52.434494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:52.434474 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:34:52.698600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:52.698515 4711 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod5b65a96d-1374-4b61-b3ed-3a12eb395abf"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod5b65a96d-1374-4b61-b3ed-3a12eb395abf] : Timed out while waiting for systemd to remove kubepods-burstable-pod5b65a96d_1374_4b61_b3ed_3a12eb395abf.slice" Nov 03 20:34:52.698600 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:34:52.698553 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod5b65a96d-1374-4b61-b3ed-3a12eb395abf] : unable to destroy cgroup paths for cgroup [kubepods burstable pod5b65a96d-1374-4b61-b3ed-3a12eb395abf] : Timed out while waiting for systemd to remove kubepods-burstable-pod5b65a96d_1374_4b61_b3ed_3a12eb395abf.slice" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" Nov 03 20:34:53.071174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:53.071142 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv" Nov 03 20:34:53.117862 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:53.117816 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv"] Nov 03 20:34:53.121709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:53.121685 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-9fbc61-openshift-default-55d484c997-7t9gv"] Nov 03 20:34:53.137205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:53.137179 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:53.927832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:53.927799 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:54.596331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:54.596295 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:54.685337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:54.685302 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b65a96d-1374-4b61-b3ed-3a12eb395abf" path="/var/lib/kubelet/pods/5b65a96d-1374-4b61-b3ed-3a12eb395abf/volumes" Nov 03 20:34:55.276544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:55.276509 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:56.119647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:56.119606 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:56.811889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:56.811854 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:57.501791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:57.501752 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:57.630116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:57.630080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:58.288874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:58.288839 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:58.420897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:58.420863 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:58.996058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:58.996022 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:34:59.978009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:34:59.977974 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:00.768320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:00.768276 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:01.497538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:01.497497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:02.229508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:02.229461 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:02.919881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:02.919846 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:03.618276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:03.618238 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:03.939298 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:03.939204 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:04.293210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:04.293153 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:05.049729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:05.049694 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:05.250218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:05.250187 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:05.984295 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:05.984262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:06.356415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:06.354558 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:06.539782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:06.539749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:07.223824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.223783 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn"] Nov 03 20:35:07.234309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.233692 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.240614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.240572 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c7aecb-serving-cert\"" Nov 03 20:35:07.252583 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.252561 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn"] Nov 03 20:35:07.309340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.309307 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l959m\" (UniqueName: \"kubernetes.io/projected/4c6a9a9c-bd88-4ce1-a661-691688449924-kube-api-access-l959m\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.309681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.309430 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-metrics-certs\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.309681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.309479 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6a9a9c-bd88-4ce1-a661-691688449924-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.309681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.309625 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.310696 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.310669 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:07.411534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.411506 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l959m\" (UniqueName: \"kubernetes.io/projected/4c6a9a9c-bd88-4ce1-a661-691688449924-kube-api-access-l959m\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.411954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.411620 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-metrics-certs\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.411954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.411640 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6a9a9c-bd88-4ce1-a661-691688449924-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.411954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.411676 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.412367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.412340 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6a9a9c-bd88-4ce1-a661-691688449924-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.414267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.414244 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.414376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.414351 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-metrics-certs\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.423521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.423493 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l959m\" (UniqueName: \"kubernetes.io/projected/4c6a9a9c-bd88-4ce1-a661-691688449924-kube-api-access-l959m\") pod \"maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.438905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.438880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:07.569856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.569810 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:07.578880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:07.578843 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:08.321866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:08.321832 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn"] Nov 03 20:35:08.723536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:08.720523 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:08.877215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:08.872353 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:09.170374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:09.170325 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:09.233612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:09.233512 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" event={"ID":"4c6a9a9c-bd88-4ce1-a661-691688449924","Type":"ContainerStarted","Data":"c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969"} Nov 03 20:35:09.233612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:09.233574 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" event={"ID":"4c6a9a9c-bd88-4ce1-a661-691688449924","Type":"ContainerStarted","Data":"84d4fd7c6f34f88786b1f699ac1b0e07d0b8b477072750061781aeda05306b52"} Nov 03 20:35:09.233612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:09.233601 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:09.282971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:09.282909 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" podStartSLOduration=1.696285124 podStartE2EDuration="2.28288976s" totalImagesPullingTime="586.604636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:09.277151316 +0000 UTC m=+2012.727257586" watchObservedRunningTime="2026-11-03 20:35:09.28288976 +0000 UTC m=+2012.732995809" Nov 03 20:35:09.900917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:09.900880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:10.027765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:10.027735 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:10.705785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:10.705747 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:10.857970 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:10.857900 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:11.267618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:11.264388 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:11.393410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:11.393382 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:11.797167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:11.797131 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:11.928743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:11.928661 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:12.315308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:12.315271 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:12.497092 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:12.497054 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:12.629336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:12.629227 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:13.601969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:13.601931 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:14.319146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:14.319109 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:15.094401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:15.094360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:15.249796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:15.249767 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 20:35:15.786027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:15.785988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:16.872888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:16.872853 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:16.995816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:16.995773 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:17.476579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.476545 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll"] Nov 03 20:35:17.482831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.482809 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.488235 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.488205 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-05473b-openshift-default-dockercfg-25bff\"" Nov 03 20:35:17.494434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.494407 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll"] Nov 03 20:35:17.546271 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:35:17.546230 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:35:17.564841 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.564803 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-token\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.565055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.564858 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87pq\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-kube-api-access-c87pq\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.565055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.564970 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.565055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.565008 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-certs\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.565192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.565092 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-credential-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.565233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.565196 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-podinfo\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.565273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.565255 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-envoy\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.565414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.565397 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-data\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.565556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.565540 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6aa508-a110-46f4-b823-c4b46e04bb55-istiod-ca-cert\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667035 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-credential-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667144 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-podinfo\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667282 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667211 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-envoy\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667289 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-data\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667333 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6aa508-a110-46f4-b823-c4b46e04bb55-istiod-ca-cert\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667466 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-token\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667499 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87pq\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-kube-api-access-c87pq\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667575 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667515 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-credential-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667575 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667535 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667580 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-certs\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667808 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667782 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-data\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667835 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-socket\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.667975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.667945 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-certs\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.668479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.668434 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6aa508-a110-46f4-b823-c4b46e04bb55-istiod-ca-cert\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.670135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.670111 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-envoy\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.670251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.670136 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-podinfo\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.680155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.680125 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-token\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.680348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.680326 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87pq\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-kube-api-access-c87pq\") pod \"e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:17.798823 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.798781 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:17.820619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:17.801751 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:18.153246 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:35:18.153213 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6aa508_a110_46f4_b823_c4b46e04bb55.slice/crio-29c90b6d19bcfea63bd077b0877147c43b71e8290f28609362d434818f528313 WatchSource:0}: Error finding container 29c90b6d19bcfea63bd077b0877147c43b71e8290f28609362d434818f528313: Status 404 returned error can't find the container with id 29c90b6d19bcfea63bd077b0877147c43b71e8290f28609362d434818f528313 Nov 03 20:35:18.153711 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:18.153248 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll"] Nov 03 20:35:18.156821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:18.156786 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:35:18.156899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:18.156875 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:35:18.156945 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:18.156914 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:35:18.428144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:18.428049 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" event={"ID":"1c6aa508-a110-46f4-b823-c4b46e04bb55","Type":"ContainerStarted","Data":"cffbf7c5077f157e401815a06e69090d4a133b08e90ba6e898afc1e12ba67470"} Nov 03 20:35:18.428144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:18.428102 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" event={"ID":"1c6aa508-a110-46f4-b823-c4b46e04bb55","Type":"ContainerStarted","Data":"29c90b6d19bcfea63bd077b0877147c43b71e8290f28609362d434818f528313"} Nov 03 20:35:18.454192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:18.454140 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" podStartSLOduration=1.454125834 podStartE2EDuration="1.454125834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:18.451185562 +0000 UTC m=+2021.901291647" watchObservedRunningTime="2026-11-03 20:35:18.454125834 +0000 UTC m=+2021.904231880" Nov 03 20:35:18.499335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:18.499295 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:18.802198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:18.802160 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:19.306512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:19.306462 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:19.803764 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:19.803734 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:35:19.803925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:19.803790 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 20:35:20.031144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:20.030019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:20.755521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:20.755466 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:20.807501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:20.806940 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:35:20.817338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:20.817297 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 20:35:21.588464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:21.588408 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:21.802439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:21.802399 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:35:21.802968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:21.802506 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 20:35:22.409682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:22.409650 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:22.803007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:22.802970 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:35:22.803418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:22.803024 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 20:35:23.113246 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:23.113134 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:23.802375 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:23.802328 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Nov 03 20:35:23.802624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:23.802398 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Nov 03 20:35:23.899661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:23.899627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:24.583342 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:24.583305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:24.807676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:24.807649 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:24.808070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:24.808044 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:24.808817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:24.808802 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:35:25.418901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:25.418863 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:26.108665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:26.108631 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:26.903089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:26.903052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:27.618762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:27.618720 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:28.390921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:28.390879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.233478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:29.222400 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.363376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:29.363339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:29.796709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:29.796673 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:30.500079 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:30.500041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:30.977490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:30.977420 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:31.408137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:31.408095 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:32.200351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:32.200309 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:32.363508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:32.361941 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:32.599532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:32.599494 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:32.764639 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:32.764603 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:32.917177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:32.917093 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:33.229838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:33.229748 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:33.691908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:33.691865 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:33.827385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:33.827331 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:34.394747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:34.394717 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:35.087198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:35.087161 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:35.879285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:35.879250 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:36.586300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:36.586260 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:37.319565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:37.319532 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:38.095270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:38.095220 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:38.757323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:38.757279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:39.427676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:39.427641 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:40.185868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:40.185825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:40.911975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:40.911913 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:41.705567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:41.705523 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:42.421036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:42.421001 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:42.552617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:42.552578 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:43.112577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:43.112518 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:43.755291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:43.755250 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:44.502169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:44.502127 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:45.199523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:45.199486 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:45.863675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:45.863639 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:46.711293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:46.711240 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:47.534185 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:35:47.534144 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:35:47.605436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:47.605403 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:48.403198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:48.403161 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:49.163380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:49.163327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:50.028355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:50.028316 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:50.962889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:50.962834 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:52.212025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:52.211987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:54.119922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:54.119884 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:54.455904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:54.455823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:55.225069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:55.225034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:55.357577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:55.357506 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:55.909353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:55.909310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:56.361593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:56.361552 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:56.493968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:56.493933 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:56.634104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:56.634024 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:56.774211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:56.774175 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:56.913098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:56.913002 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:57.062809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:57.062778 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:57.204708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:57.204597 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:57.638298 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:57.638258 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:57.784140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:57.784106 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:58.319326 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.319279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:58.476366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.476114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:58.766770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.766735 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk"] Nov 03 20:35:58.772794 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.772765 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:58.782366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.782054 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-421604-openshift-default-dockercfg-p242f\"" Nov 03 20:35:58.785272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.785226 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk"] Nov 03 20:35:58.960350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.960313 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581b6db0-0753-4059-8a03-e60bf2c3913b-istiod-ca-cert\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:58.960533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.960366 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-data\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:58.960602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.960585 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-certs\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:58.960645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.960621 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:58.960709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.960697 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-podinfo\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:58.960774 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.960750 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz4vb\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-kube-api-access-bz4vb\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:58.960811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.960791 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-token\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:58.961018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.961002 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-credential-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:58.961053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:58.961045 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-envoy\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.062916 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-certs\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.062977 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063035 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-podinfo\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063068 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz4vb\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-kube-api-access-bz4vb\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063096 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-token\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063322 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-credential-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063327 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-certs\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063383 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063440 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-envoy\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063511 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581b6db0-0753-4059-8a03-e60bf2c3913b-istiod-ca-cert\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063559 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-data\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.063759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063667 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-credential-socket\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.064030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.063981 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-data\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.064176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.064157 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581b6db0-0753-4059-8a03-e60bf2c3913b-istiod-ca-cert\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.065740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.065717 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-podinfo\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.065836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.065778 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-envoy\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.075436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.075412 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-token\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.075590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.075573 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz4vb\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-kube-api-access-bz4vb\") pod \"e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.091700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.091677 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:35:59.209120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.209080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:35:59.382776 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:35:59.382744 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581b6db0_0753_4059_8a03_e60bf2c3913b.slice/crio-47b7e1acc6f59ab9f03793115cf1a81870452c95863e95a99ac3a6ff17baf855 WatchSource:0}: Error finding container 47b7e1acc6f59ab9f03793115cf1a81870452c95863e95a99ac3a6ff17baf855: Status 404 returned error can't find the container with id 47b7e1acc6f59ab9f03793115cf1a81870452c95863e95a99ac3a6ff17baf855 Nov 03 20:35:59.383075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.383048 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk"] Nov 03 20:35:59.387039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.386999 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:35:59.387137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.387088 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:35:59.387205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.387146 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:35:59.935076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:35:59.935043 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:00.107905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:00.107860 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" event={"ID":"581b6db0-0753-4059-8a03-e60bf2c3913b","Type":"ContainerStarted","Data":"791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8"} Nov 03 20:36:00.108077 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:00.107915 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" event={"ID":"581b6db0-0753-4059-8a03-e60bf2c3913b","Type":"ContainerStarted","Data":"47b7e1acc6f59ab9f03793115cf1a81870452c95863e95a99ac3a6ff17baf855"} Nov 03 20:36:00.136066 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:00.136012 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" podStartSLOduration=2.135997698 podStartE2EDuration="2.135997698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:00.130831916 +0000 UTC m=+2063.580937977" watchObservedRunningTime="2026-11-03 20:36:00.135997698 +0000 UTC m=+2063.586103743" Nov 03 20:36:00.620860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:00.620822 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:01.093485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:01.092163 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:36:01.094197 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:01.094173 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 20:36:01.094278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:01.094223 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 20:36:01.365121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:01.357890 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:01.978998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:01.978962 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:02.097473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:02.092724 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 20:36:02.097473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:02.092792 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 20:36:02.715163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:02.715125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:03.095815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:03.095777 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 20:36:03.096041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:03.095841 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 20:36:03.510388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:03.510349 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:04.092955 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:04.092921 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 20:36:04.093167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:04.092979 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 20:36:04.297323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:04.297286 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:04.963597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:04.963551 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:05.092468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:05.092078 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 20:36:05.092468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:05.092137 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 20:36:05.719584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:05.719551 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:06.092860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:06.092825 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Nov 03 20:36:06.093347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:06.092878 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Nov 03 20:36:06.402952 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:06.402845 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:07.097977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:07.097947 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:36:07.098461 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:07.098428 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:36:07.099053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:07.099037 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 20:36:07.198137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:07.198103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:07.918530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:07.918492 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:08.971155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:08.971120 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:10.326151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:10.326116 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:11.200299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:11.200260 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:11.912045 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:11.912014 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:12.688290 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:12.688256 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:12.819631 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:12.819595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:12.958732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:12.958615 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:13.253435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:13.253350 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:13.399669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:13.399639 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:13.538871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:13.538831 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:13.719593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:13.719554 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:13.866210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:13.866114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:14.045334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:14.045300 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:14.332468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:14.325152 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:14.744014 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:14.743930 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:15.477341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:15.477302 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:15.613222 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:15.613177 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:15.752573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:15.752476 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:15.905538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:15.905501 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:16.050813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.050777 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:16.824071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.824036 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74"] Nov 03 20:36:16.829102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.829081 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:16.836360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.836335 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"payload-processing-external-model-plugins-e2e-extmod-b-03c633\"" Nov 03 20:36:16.836360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.836335 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"agc-cred-371723960661\"" Nov 03 20:36:16.837068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.837044 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"agc-cred-d84007a46645\"" Nov 03 20:36:16.837133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.837118 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"kube-root-ca.crt\"" Nov 03 20:36:16.837193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.837178 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"routing-overlay\"" Nov 03 20:36:16.837333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.837318 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"payload-processing-external-model-e2e--bb65a178-dockercfg-5zp5h\"" Nov 03 20:36:16.837395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.837370 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-03c633\"/\"openshift-service-ca.crt\"" Nov 03 20:36:16.839391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.838781 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74"] Nov 03 20:36:16.914336 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.914296 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:16.914613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.914591 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98vt\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-kube-api-access-r98vt\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:16.914798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.914781 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:16.914913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.914898 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:16.967346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:16.967316 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:17.030084 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.029941 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:17.031540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.031512 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:17.054840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.054798 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98vt\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-kube-api-access-r98vt\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:17.056134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.056093 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:17.057094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.055813 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:17.057355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.031418 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:17.079993 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.078251 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98vt\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-kube-api-access-r98vt\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:17.079993 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.079026 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:17.146497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.146461 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:17.437044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.436958 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74"] Nov 03 20:36:17.441699 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:36:17.441672 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465033c1_002a_4676_b631_88c2cf5048fa.slice/crio-88bf02038767dd22dc7003818ce32d570de8c4bba0af2d27aed015ecccc6531e WatchSource:0}: Error finding container 88bf02038767dd22dc7003818ce32d570de8c4bba0af2d27aed015ecccc6531e: Status 404 returned error can't find the container with id 88bf02038767dd22dc7003818ce32d570de8c4bba0af2d27aed015ecccc6531e Nov 03 20:36:17.466936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.466897 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" event={"ID":"465033c1-002a-4676-b631-88c2cf5048fa","Type":"ContainerStarted","Data":"88bf02038767dd22dc7003818ce32d570de8c4bba0af2d27aed015ecccc6531e"} Nov 03 20:36:17.535073 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:36:17.535038 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:36:17.708674 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:17.708591 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:18.420488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:18.412714 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:18.479660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:18.479625 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" event={"ID":"465033c1-002a-4676-b631-88c2cf5048fa","Type":"ContainerStarted","Data":"6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12"} Nov 03 20:36:18.479865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:18.479755 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:18.507088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:18.507038 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" podStartSLOduration=2.507025111 podStartE2EDuration="2.507025111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:18.502061119 +0000 UTC m=+2081.952167166" watchObservedRunningTime="2026-11-03 20:36:18.507025111 +0000 UTC m=+2081.957131156" Nov 03 20:36:19.222376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:19.222332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:19.998101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:19.998068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:20.810970 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:20.805872 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:21.809020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:21.808984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:21.954751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:21.954713 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:22.750901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:22.750860 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:22.880557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:22.877435 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:23.401491 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:23.400001 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:24.163593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:24.163559 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:24.905279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:24.905249 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:25.619415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:25.619380 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:26.349727 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:26.349687 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:27.008434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:27.008396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:27.726367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:27.726336 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:28.410016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:28.409975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:29.098998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:29.098954 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:29.490167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:29.490097 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:36:29.878708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:29.878670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:30.533153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:30.533113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:31.513234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:31.513189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:32.616009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:32.615977 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:33.532252 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:33.532214 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:34.364471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:34.361064 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:35.220738 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:35.220695 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:36.120232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:36.120186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:36.975876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:36.975815 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:37.101900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:37.101850 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:37.241971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:37.241894 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:36:37.245975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:37.245949 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:36:37.246281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:37.246227 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:36:37.254423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:37.253709 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:36:37.981869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:37.981841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:38.791541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:38.790522 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:39.494676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:39.494637 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:40.214963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:40.214925 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:40.894310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:40.894270 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:41.658214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:41.658180 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:42.411217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:42.411180 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:43.148467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:43.146058 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:43.898012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:43.897978 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:44.708440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:44.708402 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:45.495319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:45.495272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:46.303981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:46.303947 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:47.013971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:47.013933 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:47.537848 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:36:47.537805 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:36:47.851348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:47.851255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:48.393508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:48.393471 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:48.538542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:48.528429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:49.045149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:49.045120 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:49.174586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:49.174553 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:50.087123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:50.087081 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:50.874908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:50.874863 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:51.545427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:51.545393 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:52.223368 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:52.223327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:52.911795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:52.911759 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:53.656578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:53.656542 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:54.326353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:54.326314 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:55.090772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:55.090736 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:55.440842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:55.440725 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:55.569914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:55.569879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:55.699680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:55.699565 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:55.866775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:55.866737 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:56.501075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:56.501041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:57.316988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:57.316956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:57.790923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:57.790895 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:58.094266 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:58.094186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:58.481902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:58.481796 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:58.798157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:58.798119 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:36:59.549466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:36:59.547472 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:00.236470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:00.235750 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:00.997966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:00.997923 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:01.718548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:01.718510 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:02.068435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:02.068399 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:02.202061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:02.202019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:02.506277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:02.506228 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:02.647117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:02.647039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:02.791902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:02.791755 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:03.166852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:03.164082 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:04.111894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:04.111843 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:04.839396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:04.838125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:05.653263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:05.653221 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:06.998645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:06.998592 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:07.806702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:07.806672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:08.545180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:08.545144 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:08.680806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:08.680753 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:09.399723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:09.399677 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:09.538937 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:09.538899 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:10.123285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:10.123248 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:10.254044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:10.254010 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:10.761684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:10.761651 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:11.514990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:11.514951 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:12.196704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:12.196674 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:12.964015 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:12.963973 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:13.764669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:13.764629 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:14.301703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:14.301421 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:14.442260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:14.442226 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:15.156716 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:15.156685 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:15.348153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:15.348114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:16.220748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:16.220706 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:16.906806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:16.906763 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:17.101543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:17.101512 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:17.230500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:17.230322 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:17.534197 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:37:17.534155 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:37:17.999961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:17.999872 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:18.708464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:18.708420 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:19.395525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:19.395486 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:20.228058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:20.228027 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:21.003163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:21.003125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:21.146033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:21.145988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:21.277999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:21.277968 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:21.418222 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:21.418171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:21.798943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:21.798906 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:22.076229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:22.076142 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74"] Nov 03 20:37:22.518022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:22.517984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:23.353737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:23.353700 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:24.036999 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:24.036961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:24.713804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:24.713771 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:25.391105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:25.391067 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:26.226220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:26.226172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:26.477629 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:26.477547 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:26.623470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:26.622996 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:27.077540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.077478 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" podUID="465033c1-002a-4676-b631-88c2cf5048fa" containerName="payload-processing" containerID="cri-o://6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12" gracePeriod=25 Nov 03 20:37:27.382662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.382630 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:37:27.475041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.474990 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-kube-api-access-r98vt\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-kube-api-access-r98vt\") pod \"465033c1-002a-4676-b631-88c2cf5048fa\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " Nov 03 20:37:27.475355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.475335 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-provider-credentials\") pod \"465033c1-002a-4676-b631-88c2cf5048fa\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " Nov 03 20:37:27.475552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.475516 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-plugins-config-volume\") pod \"465033c1-002a-4676-b631-88c2cf5048fa\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " Nov 03 20:37:27.475734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.475715 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-routing-overlay-volume\") pod \"465033c1-002a-4676-b631-88c2cf5048fa\" (UID: \"465033c1-002a-4676-b631-88c2cf5048fa\") " Nov 03 20:37:27.476144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.476114 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-plugins-config-volume" pod "465033c1-002a-4676-b631-88c2cf5048fa" (UID: "465033c1-002a-4676-b631-88c2cf5048fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:27.476238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.476144 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-routing-overlay-volume" pod "465033c1-002a-4676-b631-88c2cf5048fa" (UID: "465033c1-002a-4676-b631-88c2cf5048fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:27.477054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.477030 4711 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-routing-overlay-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:27.477196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.477181 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/465033c1-002a-4676-b631-88c2cf5048fa-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:27.479771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.479720 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-provider-credentials" pod "465033c1-002a-4676-b631-88c2cf5048fa" (UID: "465033c1-002a-4676-b631-88c2cf5048fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:27.481080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.481049 4711 generic.go:356] "Generic (PLEG): container finished" podID="465033c1-002a-4676-b631-88c2cf5048fa" containerID="6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12" exitCode=0 Nov 03 20:37:27.481202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.481102 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" event={"ID":"465033c1-002a-4676-b631-88c2cf5048fa","Type":"ContainerDied","Data":"6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12"} Nov 03 20:37:27.481202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.481189 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" event={"ID":"465033c1-002a-4676-b631-88c2cf5048fa","Type":"ContainerDied","Data":"88bf02038767dd22dc7003818ce32d570de8c4bba0af2d27aed015ecccc6531e"} Nov 03 20:37:27.481321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.481221 4711 scope.go:122] "RemoveContainer" containerID="6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12" Nov 03 20:37:27.481321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.481252 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-kube-api-access-r98vt" pod "465033c1-002a-4676-b631-88c2cf5048fa" (UID: "465033c1-002a-4676-b631-88c2cf5048fa"). InnerVolumeSpecName "kube-api-access-r98vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:27.481414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.481382 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74" Nov 03 20:37:27.532249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.532223 4711 scope.go:122] "RemoveContainer" containerID="6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12" Nov 03 20:37:27.532657 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:37:27.532632 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12\": container with ID starting with 6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12 not found: ID does not exist" containerID="6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12" Nov 03 20:37:27.532747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.532669 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12"} err="failed to get container status \"6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12\": rpc error: code = NotFound desc = could not find container \"6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12\": container with ID starting with 6a2fef58a469ba7d43f33c7a2c015008ccae166a4045ff59844e1c8e0a9f6c12 not found: ID does not exist" Nov 03 20:37:27.543380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.543350 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74"] Nov 03 20:37:27.548025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.547997 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-03c633/payload-processing-external-model-e2e-extmod-b-03c633-7bdcckt74"] Nov 03 20:37:27.578556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.578530 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r98vt\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-kube-api-access-r98vt\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:27.578556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.578555 4711 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/465033c1-002a-4676-b631-88c2cf5048fa-provider-credentials\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:27.679221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.679134 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:27.911521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:27.911487 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:28.575069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:28.575032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:28.684895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:28.684861 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465033c1-002a-4676-b631-88c2cf5048fa" path="/var/lib/kubelet/pods/465033c1-002a-4676-b631-88c2cf5048fa/volumes" Nov 03 20:37:29.003220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:29.003122 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:29.146468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:29.146076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:29.354666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:29.354622 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:29.486811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:29.486775 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:29.716110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:29.716016 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:30.484127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:30.484092 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:30.639905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:30.639872 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv"] Nov 03 20:37:30.640112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:30.640082 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" containerID="cri-o://87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824" gracePeriod=30 Nov 03 20:37:31.224220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:31.224183 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:31.905537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:31.905505 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz"] Nov 03 20:37:31.905801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:31.905773 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" containerID="cri-o://f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957" gracePeriod=30 Nov 03 20:37:31.993893 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:31.993857 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:32.795738 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:32.795701 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:33.615414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:33.615376 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:33.795363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:33.795320 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:33.934352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:33.934259 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:34.409694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:34.409638 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:35.067236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:35.067183 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:35.806684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:35.806646 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:36.003453 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.003414 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:37:36.113558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.113024 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-envoy\") pod \"21270674-676b-4739-b6ed-dda82d8b1a22\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " Nov 03 20:37:36.114029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.114004 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-data" pod "21270674-676b-4739-b6ed-dda82d8b1a22" (UID: "21270674-676b-4739-b6ed-dda82d8b1a22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.114123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.114067 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-data\") pod \"21270674-676b-4739-b6ed-dda82d8b1a22\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " Nov 03 20:37:36.114701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.114678 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21270674-676b-4739-b6ed-dda82d8b1a22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21270674-676b-4739-b6ed-dda82d8b1a22-istio-podinfo\") pod \"21270674-676b-4739-b6ed-dda82d8b1a22\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " Nov 03 20:37:36.115052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.114943 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-certs\") pod \"21270674-676b-4739-b6ed-dda82d8b1a22\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " Nov 03 20:37:36.115388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.115369 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-credential-socket\") pod \"21270674-676b-4739-b6ed-dda82d8b1a22\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " Nov 03 20:37:36.115721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.115233 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-certs" pod "21270674-676b-4739-b6ed-dda82d8b1a22" (UID: "21270674-676b-4739-b6ed-dda82d8b1a22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.115721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.115656 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-credential-socket" pod "21270674-676b-4739-b6ed-dda82d8b1a22" (UID: "21270674-676b-4739-b6ed-dda82d8b1a22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.115905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.115777 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-socket\") pod \"21270674-676b-4739-b6ed-dda82d8b1a22\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " Nov 03 20:37:36.115905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.115820 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-kube-api-access-ml8xt\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-kube-api-access-ml8xt\") pod \"21270674-676b-4739-b6ed-dda82d8b1a22\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " Nov 03 20:37:36.116019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.115937 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-istio-token\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-istio-token\") pod \"21270674-676b-4739-b6ed-dda82d8b1a22\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " Nov 03 20:37:36.116076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.116044 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21270674-676b-4739-b6ed-dda82d8b1a22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21270674-676b-4739-b6ed-dda82d8b1a22-istiod-ca-cert\") pod \"21270674-676b-4739-b6ed-dda82d8b1a22\" (UID: \"21270674-676b-4739-b6ed-dda82d8b1a22\") " Nov 03 20:37:36.116958 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.116923 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-socket" pod "21270674-676b-4739-b6ed-dda82d8b1a22" (UID: "21270674-676b-4739-b6ed-dda82d8b1a22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.117366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.117342 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21270674-676b-4739-b6ed-dda82d8b1a22-istiod-ca-cert" pod "21270674-676b-4739-b6ed-dda82d8b1a22" (UID: "21270674-676b-4739-b6ed-dda82d8b1a22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:36.121463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.118982 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-istio-token" pod "21270674-676b-4739-b6ed-dda82d8b1a22" (UID: "21270674-676b-4739-b6ed-dda82d8b1a22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:36.121463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.119206 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.121463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.119231 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.121463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.119247 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.121463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.119261 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.121463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.119274 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.121463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.119287 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21270674-676b-4739-b6ed-dda82d8b1a22-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.134080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.134023 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21270674-676b-4739-b6ed-dda82d8b1a22-istio-podinfo" pod "21270674-676b-4739-b6ed-dda82d8b1a22" (UID: "21270674-676b-4739-b6ed-dda82d8b1a22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:36.134652 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.134608 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-envoy" pod "21270674-676b-4739-b6ed-dda82d8b1a22" (UID: "21270674-676b-4739-b6ed-dda82d8b1a22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:36.138122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.138083 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-kube-api-access-ml8xt" pod "21270674-676b-4739-b6ed-dda82d8b1a22" (UID: "21270674-676b-4739-b6ed-dda82d8b1a22"). InnerVolumeSpecName "kube-api-access-ml8xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:36.211157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.210965 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:36.229921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.225502 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8xt\" (UniqueName: \"kubernetes.io/projected/21270674-676b-4739-b6ed-dda82d8b1a22-kube-api-access-ml8xt\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.229921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.225546 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21270674-676b-4739-b6ed-dda82d8b1a22-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.229921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.225563 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21270674-676b-4739-b6ed-dda82d8b1a22-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:36.321572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.319562 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6"] Nov 03 20:37:36.321572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.319854 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" containerID="cri-o://fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83" gracePeriod=30 Nov 03 20:37:36.364172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.364131 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:36.494316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.494227 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:36.612140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.612104 4711 generic.go:356] "Generic (PLEG): container finished" podID="21270674-676b-4739-b6ed-dda82d8b1a22" containerID="87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824" exitCode=0 Nov 03 20:37:36.612140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.612140 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" event={"ID":"21270674-676b-4739-b6ed-dda82d8b1a22","Type":"ContainerDied","Data":"87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824"} Nov 03 20:37:36.612347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.612184 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" Nov 03 20:37:36.612347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.612195 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv" event={"ID":"21270674-676b-4739-b6ed-dda82d8b1a22","Type":"ContainerDied","Data":"b58d0b26b16f91d642685421e1a91641c17298bb30e213a73b3fe90ed57d3d61"} Nov 03 20:37:36.612347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.612223 4711 scope.go:122] "RemoveContainer" containerID="87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824" Nov 03 20:37:36.623800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.623779 4711 scope.go:122] "RemoveContainer" containerID="87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824" Nov 03 20:37:36.624077 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:37:36.624051 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824\": container with ID starting with 87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824 not found: ID does not exist" containerID="87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824" Nov 03 20:37:36.624119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.624086 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824"} err="failed to get container status \"87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824\": rpc error: code = NotFound desc = could not find container \"87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824\": container with ID starting with 87c1f79cdd46fd189e4afce87cf04f9d05cde2b9acb0c388a1bef044f86bd824 not found: ID does not exist" Nov 03 20:37:36.638543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.638515 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv"] Nov 03 20:37:36.643997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.643974 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-546kv"] Nov 03 20:37:36.682903 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.682869 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" path="/var/lib/kubelet/pods/21270674-676b-4739-b6ed-dda82d8b1a22/volumes" Nov 03 20:37:36.712471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:36.710433 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:37.218621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.218589 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:37:37.268713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.268679 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eef084f-fa41-48aa-becf-55d52edd3050-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eef084f-fa41-48aa-becf-55d52edd3050-istiod-ca-cert\") pod \"1eef084f-fa41-48aa-becf-55d52edd3050\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " Nov 03 20:37:37.268917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.268787 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1eef084f-fa41-48aa-becf-55d52edd3050-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eef084f-fa41-48aa-becf-55d52edd3050-istio-podinfo\") pod \"1eef084f-fa41-48aa-becf-55d52edd3050\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " Nov 03 20:37:37.268917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.268846 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-data\") pod \"1eef084f-fa41-48aa-becf-55d52edd3050\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " Nov 03 20:37:37.268917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.268879 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-envoy\") pod \"1eef084f-fa41-48aa-becf-55d52edd3050\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " Nov 03 20:37:37.269041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.268951 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-kube-api-access-pvwx8\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-kube-api-access-pvwx8\") pod \"1eef084f-fa41-48aa-becf-55d52edd3050\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " Nov 03 20:37:37.269041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.268987 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-istio-token\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-istio-token\") pod \"1eef084f-fa41-48aa-becf-55d52edd3050\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " Nov 03 20:37:37.269126 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.269105 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eef084f-fa41-48aa-becf-55d52edd3050-istiod-ca-cert" pod "1eef084f-fa41-48aa-becf-55d52edd3050" (UID: "1eef084f-fa41-48aa-becf-55d52edd3050"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:37.269183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.269131 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-socket\") pod \"1eef084f-fa41-48aa-becf-55d52edd3050\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " Nov 03 20:37:37.269604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.269259 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-credential-socket\") pod \"1eef084f-fa41-48aa-becf-55d52edd3050\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " Nov 03 20:37:37.269604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.269382 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-certs\") pod \"1eef084f-fa41-48aa-becf-55d52edd3050\" (UID: \"1eef084f-fa41-48aa-becf-55d52edd3050\") " Nov 03 20:37:37.269604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.269483 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-data" pod "1eef084f-fa41-48aa-becf-55d52edd3050" (UID: "1eef084f-fa41-48aa-becf-55d52edd3050"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:37.269872 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.269734 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-socket" pod "1eef084f-fa41-48aa-becf-55d52edd3050" (UID: "1eef084f-fa41-48aa-becf-55d52edd3050"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:37.270279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.270157 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-credential-socket" pod "1eef084f-fa41-48aa-becf-55d52edd3050" (UID: "1eef084f-fa41-48aa-becf-55d52edd3050"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:37.270279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.270229 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-certs" pod "1eef084f-fa41-48aa-becf-55d52edd3050" (UID: "1eef084f-fa41-48aa-becf-55d52edd3050"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:37.270467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.270433 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:37.270537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.270472 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:37.270537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.270489 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:37.270537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.270502 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1eef084f-fa41-48aa-becf-55d52edd3050-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:37.270537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.270515 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:37.271512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.271488 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-envoy" pod "1eef084f-fa41-48aa-becf-55d52edd3050" (UID: "1eef084f-fa41-48aa-becf-55d52edd3050"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:37.273154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.272334 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-istio-token" pod "1eef084f-fa41-48aa-becf-55d52edd3050" (UID: "1eef084f-fa41-48aa-becf-55d52edd3050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:37.273396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.273373 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1eef084f-fa41-48aa-becf-55d52edd3050-istio-podinfo" pod "1eef084f-fa41-48aa-becf-55d52edd3050" (UID: "1eef084f-fa41-48aa-becf-55d52edd3050"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:37.273619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.273545 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-kube-api-access-pvwx8" pod "1eef084f-fa41-48aa-becf-55d52edd3050" (UID: "1eef084f-fa41-48aa-becf-55d52edd3050"). InnerVolumeSpecName "kube-api-access-pvwx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:37.385496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.365674 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:37.385496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.382134 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1eef084f-fa41-48aa-becf-55d52edd3050-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:37.418652 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.418620 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1eef084f-fa41-48aa-becf-55d52edd3050-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:37.418867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.418676 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvwx8\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-kube-api-access-pvwx8\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:37.418867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.418693 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1eef084f-fa41-48aa-becf-55d52edd3050-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:37.621914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.621884 4711 generic.go:356] "Generic (PLEG): container finished" podID="1eef084f-fa41-48aa-becf-55d52edd3050" containerID="f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957" exitCode=0 Nov 03 20:37:37.622129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.621929 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" Nov 03 20:37:37.622129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.621968 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" event={"ID":"1eef084f-fa41-48aa-becf-55d52edd3050","Type":"ContainerDied","Data":"f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957"} Nov 03 20:37:37.622129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.622016 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz" event={"ID":"1eef084f-fa41-48aa-becf-55d52edd3050","Type":"ContainerDied","Data":"47d4de77c9cab2293bc29517865ac78a9ebed315401a1bd88dd59b60a9f2618f"} Nov 03 20:37:37.622129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.622043 4711 scope.go:122] "RemoveContainer" containerID="f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957" Nov 03 20:37:37.636626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.636607 4711 scope.go:122] "RemoveContainer" containerID="f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957" Nov 03 20:37:37.636941 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:37:37.636916 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957\": container with ID starting with f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957 not found: ID does not exist" containerID="f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957" Nov 03 20:37:37.637033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.636948 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957"} err="failed to get container status \"f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957\": rpc error: code = NotFound desc = could not find container \"f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957\": container with ID starting with f7af3a552cd6cccb296dde5dc92a6b0285a28977e420b4a824a4380eb20cc957 not found: ID does not exist" Nov 03 20:37:37.651760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.651737 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz"] Nov 03 20:37:37.655854 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:37.655835 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-9thpz"] Nov 03 20:37:38.201044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:38.201002 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:38.683155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:38.683123 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" path="/var/lib/kubelet/pods/1eef084f-fa41-48aa-becf-55d52edd3050/volumes" Nov 03 20:37:38.889516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:38.889483 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:39.592582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:39.592542 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:40.434947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:40.426114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:41.082414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.082347 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:41.620904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.620874 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:37:41.655397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.655368 4711 generic.go:356] "Generic (PLEG): container finished" podID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerID="fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83" exitCode=0 Nov 03 20:37:41.655596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.655424 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" Nov 03 20:37:41.655596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.655460 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" event={"ID":"5d0a6f21-836d-4ba3-b99c-025e142e4c01","Type":"ContainerDied","Data":"fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83"} Nov 03 20:37:41.655596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.655517 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6" event={"ID":"5d0a6f21-836d-4ba3-b99c-025e142e4c01","Type":"ContainerDied","Data":"a55c6359bb13af6af8e93859013da44411986260145402c47a59c32bc4c725db"} Nov 03 20:37:41.655596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.655545 4711 scope.go:122] "RemoveContainer" containerID="fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83" Nov 03 20:37:41.666199 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.666175 4711 scope.go:122] "RemoveContainer" containerID="fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83" Nov 03 20:37:41.666556 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:37:41.666529 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83\": container with ID starting with fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83 not found: ID does not exist" containerID="fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83" Nov 03 20:37:41.666660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.666570 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83"} err="failed to get container status \"fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83\": rpc error: code = NotFound desc = could not find container \"fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83\": container with ID starting with fc46f746a9cffe03a8424a7fd205a1aca0847349dbd79e248e804b5e97c7cd83 not found: ID does not exist" Nov 03 20:37:41.720362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.720327 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r"] Nov 03 20:37:41.720562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.720530 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" containerID="cri-o://9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3" gracePeriod=30 Nov 03 20:37:41.784587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.784554 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:41.796604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.796575 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-token\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-token\") pod \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " Nov 03 20:37:41.796760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.796617 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istiod-ca-cert\") pod \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " Nov 03 20:37:41.796760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.796680 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-envoy\") pod \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " Nov 03 20:37:41.796890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.796833 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-socket\") pod \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " Nov 03 20:37:41.796954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.796921 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-credential-socket\") pod \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " Nov 03 20:37:41.797014 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.796995 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-data\") pod \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " Nov 03 20:37:41.797076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.797024 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-kube-api-access-2glcz\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-kube-api-access-2glcz\") pod \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " Nov 03 20:37:41.797195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.797111 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istiod-ca-cert" pod "5d0a6f21-836d-4ba3-b99c-025e142e4c01" (UID: "5d0a6f21-836d-4ba3-b99c-025e142e4c01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:41.797265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.797197 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-certs\") pod \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " Nov 03 20:37:41.797265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.797252 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-podinfo\") pod \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\" (UID: \"5d0a6f21-836d-4ba3-b99c-025e142e4c01\") " Nov 03 20:37:41.797391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.797178 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-socket" pod "5d0a6f21-836d-4ba3-b99c-025e142e4c01" (UID: "5d0a6f21-836d-4ba3-b99c-025e142e4c01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:41.797479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.797367 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-data" pod "5d0a6f21-836d-4ba3-b99c-025e142e4c01" (UID: "5d0a6f21-836d-4ba3-b99c-025e142e4c01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:41.797641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.797515 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-credential-socket" pod "5d0a6f21-836d-4ba3-b99c-025e142e4c01" (UID: "5d0a6f21-836d-4ba3-b99c-025e142e4c01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:41.797641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.797598 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-certs" pod "5d0a6f21-836d-4ba3-b99c-025e142e4c01" (UID: "5d0a6f21-836d-4ba3-b99c-025e142e4c01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:41.798232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.798212 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:41.798340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.798234 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:41.798340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.798249 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:41.798340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.798262 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:41.798340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.798278 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:41.799517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.799489 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-kube-api-access-2glcz" pod "5d0a6f21-836d-4ba3-b99c-025e142e4c01" (UID: "5d0a6f21-836d-4ba3-b99c-025e142e4c01"). InnerVolumeSpecName "kube-api-access-2glcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:41.799636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.799527 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-token" pod "5d0a6f21-836d-4ba3-b99c-025e142e4c01" (UID: "5d0a6f21-836d-4ba3-b99c-025e142e4c01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:41.799718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.799677 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-envoy" pod "5d0a6f21-836d-4ba3-b99c-025e142e4c01" (UID: "5d0a6f21-836d-4ba3-b99c-025e142e4c01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:41.800237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.800219 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-podinfo" pod "5d0a6f21-836d-4ba3-b99c-025e142e4c01" (UID: "5d0a6f21-836d-4ba3-b99c-025e142e4c01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:41.902530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.902063 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:41.902530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.902106 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:41.902530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.902120 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2glcz\" (UniqueName: \"kubernetes.io/projected/5d0a6f21-836d-4ba3-b99c-025e142e4c01-kube-api-access-2glcz\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:41.902530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:41.902135 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d0a6f21-836d-4ba3-b99c-025e142e4c01-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:42.079466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:42.078532 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6"] Nov 03 20:37:42.104467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:42.100698 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-rnvr6"] Nov 03 20:37:42.614438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:42.614395 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:42.697876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:42.697839 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" path="/var/lib/kubelet/pods/5d0a6f21-836d-4ba3-b99c-025e142e4c01/volumes" Nov 03 20:37:42.758311 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:42.758275 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:42.895134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:42.895053 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:43.134782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:43.134741 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:43.273641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:43.273608 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:43.416504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:43.416435 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:43.549335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:43.549243 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:43.680027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:43.679985 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:43.812519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:43.812393 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:44.219166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:44.219072 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:44.358425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:44.358391 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:44.980693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:44.980661 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:45.663472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:45.662924 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:46.411342 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:46.411261 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:47.024694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.024666 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:37:47.061417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061386 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-credential-socket\") pod \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " Nov 03 20:37:47.061583 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061474 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-podinfo\") pod \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " Nov 03 20:37:47.061583 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061522 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-socket\") pod \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " Nov 03 20:37:47.061583 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061576 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-envoy\") pod \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " Nov 03 20:37:47.061769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061690 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-credential-socket" pod "eefc927f-55b7-47d4-b69e-e8bb133ec73e" (UID: "eefc927f-55b7-47d4-b69e-e8bb133ec73e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.061769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061730 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-data\") pod \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " Nov 03 20:37:47.061870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061799 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-kube-api-access-pmcwg\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-kube-api-access-pmcwg\") pod \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " Nov 03 20:37:47.061870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061835 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-socket" pod "eefc927f-55b7-47d4-b69e-e8bb133ec73e" (UID: "eefc927f-55b7-47d4-b69e-e8bb133ec73e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.061979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061916 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istiod-ca-cert\") pod \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " Nov 03 20:37:47.061979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.061973 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-token\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-token\") pod \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " Nov 03 20:37:47.062082 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.062013 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-data" pod "eefc927f-55b7-47d4-b69e-e8bb133ec73e" (UID: "eefc927f-55b7-47d4-b69e-e8bb133ec73e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.062082 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.062040 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-certs\") pod \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\" (UID: \"eefc927f-55b7-47d4-b69e-e8bb133ec73e\") " Nov 03 20:37:47.062347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.062313 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-certs" pod "eefc927f-55b7-47d4-b69e-e8bb133ec73e" (UID: "eefc927f-55b7-47d4-b69e-e8bb133ec73e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.062415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.062358 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istiod-ca-cert" pod "eefc927f-55b7-47d4-b69e-e8bb133ec73e" (UID: "eefc927f-55b7-47d4-b69e-e8bb133ec73e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:37:47.063065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.063043 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.063170 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.063071 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.063170 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.063089 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.063170 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.063106 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.063170 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.063120 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.064306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.064281 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-envoy" pod "eefc927f-55b7-47d4-b69e-e8bb133ec73e" (UID: "eefc927f-55b7-47d4-b69e-e8bb133ec73e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:37:47.064629 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.064604 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-kube-api-access-pmcwg" pod "eefc927f-55b7-47d4-b69e-e8bb133ec73e" (UID: "eefc927f-55b7-47d4-b69e-e8bb133ec73e"). InnerVolumeSpecName "kube-api-access-pmcwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:47.064883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.064818 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-podinfo" pod "eefc927f-55b7-47d4-b69e-e8bb133ec73e" (UID: "eefc927f-55b7-47d4-b69e-e8bb133ec73e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:37:47.065090 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.065065 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-token" pod "eefc927f-55b7-47d4-b69e-e8bb133ec73e" (UID: "eefc927f-55b7-47d4-b69e-e8bb133ec73e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:47.164920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.164886 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.164920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.164915 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.165124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.164931 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eefc927f-55b7-47d4-b69e-e8bb133ec73e-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.165124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.164943 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmcwg\" (UniqueName: \"kubernetes.io/projected/eefc927f-55b7-47d4-b69e-e8bb133ec73e-kube-api-access-pmcwg\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:47.222458 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.222353 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:47.542506 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:37:47.542422 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:37:47.795285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.795203 4711 generic.go:356] "Generic (PLEG): container finished" podID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerID="9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3" exitCode=0 Nov 03 20:37:47.795285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.795259 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" Nov 03 20:37:47.795505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.795276 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" event={"ID":"eefc927f-55b7-47d4-b69e-e8bb133ec73e","Type":"ContainerDied","Data":"9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3"} Nov 03 20:37:47.795505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.795362 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r" event={"ID":"eefc927f-55b7-47d4-b69e-e8bb133ec73e","Type":"ContainerDied","Data":"b7386e9ff24f3128321e4206078d1bd5a2bceb64e7cc9c649d375ba955034d90"} Nov 03 20:37:47.795505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.795402 4711 scope.go:122] "RemoveContainer" containerID="9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3" Nov 03 20:37:47.807345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.807328 4711 scope.go:122] "RemoveContainer" containerID="9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3" Nov 03 20:37:47.807648 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:37:47.807626 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3\": container with ID starting with 9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3 not found: ID does not exist" containerID="9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3" Nov 03 20:37:47.807705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.807654 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3"} err="failed to get container status \"9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3\": rpc error: code = NotFound desc = could not find container \"9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3\": container with ID starting with 9230095d789fb76132eaa7648d3f40388331d602a21d12a114399eb0dd4126d3 not found: ID does not exist" Nov 03 20:37:47.822396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.822367 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r"] Nov 03 20:37:47.829022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.828997 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-xcx8r"] Nov 03 20:37:47.996163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:47.996126 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:48.682080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:48.682044 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" path="/var/lib/kubelet/pods/eefc927f-55b7-47d4-b69e-e8bb133ec73e/volumes" Nov 03 20:37:48.693788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:48.693757 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:48.826693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:48.826654 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:49.200572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:49.200406 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:49.336417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:49.335644 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:50.042193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:50.042159 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:50.182947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:50.182909 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:50.926345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:50.926305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:51.626390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:51.626356 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:52.360847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:52.360811 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:53.025828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:53.025784 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:53.817915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:53.817881 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:54.585612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:54.585572 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:54.718116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:54.718078 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:55.360238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:55.360193 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:56.128587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:56.128549 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:56.826632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:56.826594 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:57.625963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:57.625920 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:57.853211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:57.853177 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:58.396932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:58.396893 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:58.646717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:58.646682 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:59.131454 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:59.131416 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:37:59.902171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:37:59.902137 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:00.308521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:00.308491 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:00.442773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:00.442739 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:01.072667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:01.072621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:01.232062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:01.231987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:01.848804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:01.848768 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:01.983194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:01.983150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:02.142967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:02.142875 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:02.281108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:02.281071 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:02.469070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:02.468983 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:03.238335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:03.238024 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:03.892734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:03.892701 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:04.616475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:04.616410 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:05.292483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:05.292419 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:06.035295 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:06.027576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:06.698949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:06.698899 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:06.973223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:06.973134 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:07.099247 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:07.099214 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:07.226588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:07.226511 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:07.395698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:07.395666 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:08.206760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:08.206726 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:08.762947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:08.762817 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:08.901986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:08.901951 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:09.251514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:09.249426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:09.388249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:09.388142 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:09.550208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:09.550177 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:10.354501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:10.354467 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:11.113747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:11.113712 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:11.785231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:11.785191 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:12.217683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:12.217590 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:12.396495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:12.392006 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:12.961533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:12.961493 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:13.665112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:13.665071 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:13.918036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:13.917938 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:14.599026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:14.598980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:15.426769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:15.426738 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:16.196389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:16.196355 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:16.379017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:16.378985 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:16.512349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:16.512317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:16.645900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:16.645871 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:16.773394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:16.773314 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:16.916956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:16.916904 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:17.097074 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:17.096983 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:17.378428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:17.378330 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:17.536213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:17.536060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:17.566791 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:38:17.566726 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:38:17.695540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:17.695390 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:17.893404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:17.893366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:18.196035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:18.196000 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:18.331962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:18.331927 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:18.773576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:18.773518 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:18.995546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:18.995511 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:19.131521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:19.131284 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:20.108121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:20.108085 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:20.431347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:20.431250 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:20.570573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:20.570533 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:20.693925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:20.693844 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:20.947166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:20.947083 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:21.124301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:21.124268 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:21.561348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:21.561315 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:21.694043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:21.694009 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:22.306355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:22.306310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:23.092320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:23.092281 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:23.270502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:23.270426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:23.397352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:23.397257 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:23.672000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:23.671887 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:23.909216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:23.909178 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:24.428550 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:24.428509 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:24.564515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:24.563291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:25.514867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:25.514830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:27.262759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:27.262724 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:28.327145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:28.324372 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:29.028967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:29.028928 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:29.831219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:29.831181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:30.548612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:30.548562 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:31.339429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:31.339393 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:32.182094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:32.182060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:32.915031 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:32.914993 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:33.753627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:33.753591 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:34.520723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:34.519902 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:35.331025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:35.330987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:36.245362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:36.245325 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:37.145718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:37.145670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:38.036994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:38.036954 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:38.728199 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:38.728160 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:39.719192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:39.718895 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:40.501005 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:40.500969 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:41.232870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:41.232804 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:42.110042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:42.110009 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:42.938205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:42.938118 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:43.658604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:43.658572 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:44.332247 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:44.332210 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:45.102991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:45.102951 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:45.911613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:45.906693 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:46.563639 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:46.563598 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:47.310399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:47.310366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:47.533775 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:38:47.533740 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:38:47.996130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:47.996093 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:48.699671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:48.699630 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:49.427640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:49.427596 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:50.093208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:50.093168 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:50.891955 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:50.891913 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:51.752482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:51.751348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:52.398984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:52.398942 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:53.058160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:53.058128 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:53.717623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:53.717581 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:54.410494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:54.410441 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:55.063418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:55.063378 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:55.717798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:55.717697 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:56.531920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:56.531884 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:57.218291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:57.218254 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:57.906838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:57.906807 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:58.064246 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:58.062638 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:58.737471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:58.737020 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:38:59.303650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:38:59.303606 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:00.262497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:00.262460 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:01.015834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:01.015801 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:01.950584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:01.950550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:02.606513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:02.606474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:03.314534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:03.314426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:04.000204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:04.000173 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:04.683545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:04.683509 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:05.357777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:05.357725 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:06.110788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:06.110728 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:06.909404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:06.909363 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:07.596228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:07.596189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:08.417351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:08.417311 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:09.111801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:09.111764 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:09.764573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:09.764536 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:10.542091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:10.542039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:11.277873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:11.277839 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:12.113172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:12.113135 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:12.805617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:12.805582 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:13.494831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:13.494796 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:14.218423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:14.218373 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:14.895494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:14.895458 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:15.660789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:15.660739 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:16.425094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:16.424862 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:17.141751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:17.141716 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:17.533254 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:39:17.533214 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:39:17.812713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:17.812603 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:18.583399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:18.583366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:18.948213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:18.947612 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:19.104463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:19.092028 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:19.258532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:19.252223 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:19.639209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:19.639117 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:19.810797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:19.810764 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:19.991371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:19.991270 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:20.219811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:20.219769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:20.392468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:20.388033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:20.530964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:20.530927 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:21.007265 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:21.007228 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:21.138217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:21.138183 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:21.573917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:21.573882 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:21.714236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:21.714202 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:22.229325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:22.229292 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:22.471278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:22.471232 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:23.034408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:23.034373 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:23.166574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:23.166537 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:23.290303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:23.290215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:23.674927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:23.674800 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:23.813249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:23.813212 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:23.947529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:23.943868 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:24.757478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:24.757429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:25.529606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:25.529570 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:26.209595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:26.209558 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:26.990636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:26.990599 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:27.751391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:27.751360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:28.410852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:28.410813 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:29.090684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:29.090642 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:29.857828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:29.857784 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:30.541235 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:30.541199 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:31.221386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:31.221350 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:31.938301 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:31.938261 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:32.617918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:32.617879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:33.292083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:33.292029 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:34.229375 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:34.229343 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:34.905427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:34.905370 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:35.593323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:35.593282 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:36.413011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:36.412977 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:37.086415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:37.086378 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:37.763006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:37.762972 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:38.517359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:38.517318 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:39.216083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:39.216047 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:39.900944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:39.900911 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:40.663168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:40.663136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:41.421957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:41.421918 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:42.110974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:42.110935 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:42.777272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:42.777237 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:42.906287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:42.906248 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:43.558832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:43.558786 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:44.418232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:44.418188 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:45.122352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:45.122306 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:45.820959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:45.820917 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:46.504289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:46.504257 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:47.178623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:47.178584 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:47.533952 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:39:47.533913 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:39:47.915031 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:47.914933 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:48.388140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:48.388102 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:48.607415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:48.607375 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:49.295534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:49.295482 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:50.024405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:50.024367 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:50.887996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:50.887959 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:52.068393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:52.068355 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:52.871138 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:52.871102 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:53.600576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:53.600538 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:53.903109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:53.903022 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:54.054470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:54.051538 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:54.902209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:54.902174 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:55.581136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:55.581097 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:56.057229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:56.057194 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:56.791766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:56.791733 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:57.490396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:57.490359 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:57.948917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:57.948878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:58.099026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:58.098988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:58.253551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:58.241026 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:58.778830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:58.778790 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:58.915379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:58.914377 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:39:59.593279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:39:59.593247 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:00.304547 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:00.304503 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:01.089172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:01.089136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:01.673556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:01.673513 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:01.813894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:01.813846 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:01.944286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:01.944189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:02.633128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:02.633087 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:03.522644 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:03.522611 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:03.882919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:03.882828 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:04.021537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:04.021496 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:04.209432 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:04.209339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:04.998173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:04.998125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:05.337750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:05.337718 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:05.539258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:05.539224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:05.977910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:05.977873 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:06.604671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:06.604633 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:07.288783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:07.288207 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:08.012980 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:08.012940 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:08.717278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:08.717245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:09.513695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:09.513623 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:10.246343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:10.246270 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:11.126666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:11.126621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:11.873021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:11.872979 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:12.890829 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:12.890788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:13.603735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:13.601619 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:14.276705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:14.276663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:15.116417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:15.116376 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:15.767208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:15.767169 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:16.269089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:16.269055 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:16.419026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:16.418989 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:17.126589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:17.126550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:17.535690 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:40:17.535624 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:40:17.897063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:17.896962 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:18.677578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:18.677541 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:19.343714 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:19.343663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:20.107026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:20.106986 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:20.789640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:20.789601 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:21.560537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:21.560495 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:22.548364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:22.548328 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:23.304394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:23.304359 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:24.116334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:24.116301 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:24.901683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:24.901645 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:25.617960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:25.617918 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:26.322390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:26.322351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:27.029058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:27.029024 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:27.691843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:27.691807 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:28.414837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:28.414788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:29.117605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:29.117569 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:29.786993 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:29.786956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:30.526415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:30.526376 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:30.875627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:30.875544 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn"] Nov 03 20:40:30.875792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:30.875770 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" containerID="cri-o://d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e" gracePeriod=30 Nov 03 20:40:31.318683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:31.318642 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:32.041986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:32.041929 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:32.795621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:32.795578 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:33.616704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:33.616672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:34.416932 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:34.416890 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:35.105409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:35.105373 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:35.766323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:35.766289 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:36.159637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.159606 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:40:36.219159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.219124 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-podinfo\") pod \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " Nov 03 20:40:36.219355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.219321 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-kube-api-access-sw54q\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-kube-api-access-sw54q\") pod \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " Nov 03 20:40:36.219485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.219361 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-socket\") pod \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " Nov 03 20:40:36.219485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.219465 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istiod-ca-cert\") pod \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " Nov 03 20:40:36.219611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.219529 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-token\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-token\") pod \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " Nov 03 20:40:36.219611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.219587 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-data\") pod \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " Nov 03 20:40:36.219715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.219656 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-credential-socket\") pod \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " Nov 03 20:40:36.219715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.219690 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-envoy\") pod \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " Nov 03 20:40:36.219818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.219728 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-certs\") pod \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\" (UID: \"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db\") " Nov 03 20:40:36.220344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.220239 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istiod-ca-cert" pod "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" (UID: "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:36.220907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.220663 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:36.220907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.220691 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-certs" pod "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" (UID: "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:36.221389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.221090 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-socket" pod "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" (UID: "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:36.221389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.221146 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-credential-socket" pod "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" (UID: "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:36.221389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.221190 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-data" pod "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" (UID: "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:36.223191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.223166 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-envoy" pod "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" (UID: "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:36.225666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.225640 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-kube-api-access-sw54q" pod "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" (UID: "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db"). InnerVolumeSpecName "kube-api-access-sw54q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:36.226182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.226150 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-podinfo" pod "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" (UID: "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:40:36.226504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.226437 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-token" pod "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" (UID: "ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:36.323073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.322986 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:36.323073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.323016 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw54q\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-kube-api-access-sw54q\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:36.323073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.323030 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:36.323073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.323046 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:36.323073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.323059 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:36.323073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.323072 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:36.323526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.323084 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:36.323526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.323095 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:36.523034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.522970 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:36.657253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.657144 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:36.808105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.802376 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:36.883040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.883010 4711 generic.go:356] "Generic (PLEG): container finished" podID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerID="d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e" exitCode=0 Nov 03 20:40:36.883237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.883087 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" Nov 03 20:40:36.883237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.883096 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" event={"ID":"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db","Type":"ContainerDied","Data":"d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e"} Nov 03 20:40:36.883237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.883164 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn" event={"ID":"ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db","Type":"ContainerDied","Data":"edd00a89c9f74a359df44b83ac5f88aa573937b11443d5db370edbc8b50f7c8b"} Nov 03 20:40:36.883237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.883193 4711 scope.go:122] "RemoveContainer" containerID="d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e" Nov 03 20:40:36.894591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.894568 4711 scope.go:122] "RemoveContainer" containerID="d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e" Nov 03 20:40:36.894891 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:40:36.894870 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e\": container with ID starting with d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e not found: ID does not exist" containerID="d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e" Nov 03 20:40:36.894971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.894898 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e"} err="failed to get container status \"d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e\": rpc error: code = NotFound desc = could not find container \"d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e\": container with ID starting with d715d5f36e46730757b8b17347a00abf512883b08bafd587f7dabb88b8ba606e not found: ID does not exist" Nov 03 20:40:36.908508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.908431 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn"] Nov 03 20:40:36.912656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:36.912634 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ba89e0-openshift-default-797cbf55cc-8cqkn"] Nov 03 20:40:37.717713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:37.717663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:38.400542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:38.400500 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:38.683285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:38.683197 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" path="/var/lib/kubelet/pods/ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db/volumes" Nov 03 20:40:39.098647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:39.098614 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:39.899881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:39.899848 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:40.584083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:40.584053 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:41.265269 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:41.265237 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:42.001404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:42.001370 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:43.151115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:43.151081 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:44.173720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:44.173673 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:45.188868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:45.188816 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:46.391975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:46.391876 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:47.534068 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:40:47.534024 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:40:47.916920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:47.907969 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:48.551309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:48.551273 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:48.739412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:48.739375 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:49.044833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:49.044779 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:49.189470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:49.185862 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:49.536309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:49.536270 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:50.309557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:50.299280 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:51.079817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:51.079780 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:51.722798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:51.722755 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:52.531940 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:52.531907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:53.295088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:53.295040 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:54.202277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:54.202240 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:55.446828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:55.446793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:55.578574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:55.578538 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:55.719182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:55.719091 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:55.849831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:55.849788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:56.510352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:56.510315 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:57.299801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:57.299756 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:57.994435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:57.994401 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:58.123981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:58.123942 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:58.810086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:58.810052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:58.952380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:58.952344 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:59.084392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:59.084288 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:59.225716 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:59.225683 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:59.358768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:59.358672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:59.504391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:59.504358 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:40:59.696155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:40:59.696054 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:00.257034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:00.257000 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:00.733764 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:00.733673 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:00.953922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:00.953888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:01.446647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:01.446610 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:01.692775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:01.692729 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:02.406078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:02.406033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:02.561270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:02.561231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:02.747395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:02.747309 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:02.887910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:02.887877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:03.026397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:03.026305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:03.302767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:03.302677 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:03.451498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:03.451437 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:03.877906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:03.877877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:04.016471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:04.013961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:04.562305 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:04.562266 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:05.097686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:05.097654 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:05.594208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:05.594018 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:05.729374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:05.729344 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:06.062665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:06.062532 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:06.356600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:06.356513 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:06.489837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:06.489802 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:06.647143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:06.647066 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:07.399046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:07.399014 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:08.012102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:08.012067 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:08.157425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:08.157387 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:08.303377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:08.303287 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:08.517000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:08.516958 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:08.652665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:08.652591 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:09.209759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:09.209699 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:09.826773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:09.826671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:09.975228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:09.975172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:10.120020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:10.119943 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:11.004388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:11.004347 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:11.773476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:11.773000 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:12.081175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:12.081061 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:12.218472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:12.216867 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:13.180081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:13.180046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:13.316032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:13.315994 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:13.514739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:13.514700 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:14.636360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:14.636326 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:15.713520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:15.706915 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:15.862598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:15.862564 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:16.018867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:16.018838 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:16.371926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:16.371805 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:16.517734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:16.517705 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:17.040520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:17.040437 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:17.192858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:17.192804 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:17.537284 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:41:17.537249 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:41:18.022781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:18.022113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:18.174226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:18.174192 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:18.551470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:18.551324 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:19.383433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:19.383394 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:20.103537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:20.103493 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:20.926258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:20.926224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:21.692138 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:21.692095 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:22.393618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:22.393569 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:22.529436 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:22.529401 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:23.506559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:23.506511 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:23.792949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:23.792866 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:23.927939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:23.927896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:24.115437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:24.115345 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:24.390569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:24.390474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:24.524335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:24.524283 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:25.082592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.082563 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:25.232470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.230859 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:25.661298 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.661259 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:25.801243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.801210 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw"] Nov 03 20:41:25.801944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.801929 4711 cpu_manager.go:414] "containerMap: removing container" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" Nov 03 20:41:25.802016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.801945 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" Nov 03 20:41:25.802016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.801959 4711 cpu_manager.go:414] "containerMap: removing container" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" Nov 03 20:41:25.802016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.801964 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" Nov 03 20:41:25.802016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.801983 4711 cpu_manager.go:414] "containerMap: removing container" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" Nov 03 20:41:25.802016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.801988 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" Nov 03 20:41:25.802016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802010 4711 cpu_manager.go:414] "containerMap: removing container" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" Nov 03 20:41:25.802016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802015 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802024 4711 cpu_manager.go:414] "containerMap: removing container" podUID="465033c1-002a-4676-b631-88c2cf5048fa" containerName="payload-processing" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802028 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465033c1-002a-4676-b631-88c2cf5048fa" containerName="payload-processing" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802039 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802045 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802153 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="21270674-676b-4739-b6ed-dda82d8b1a22" containerName="istio-proxy" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802161 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="465033c1-002a-4676-b631-88c2cf5048fa" containerName="payload-processing" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802168 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2a83b1-1b4a-4624-b0c6-bdb4a275d7db" containerName="istio-proxy" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802178 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0a6f21-836d-4ba3-b99c-025e142e4c01" containerName="istio-proxy" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802183 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefc927f-55b7-47d4-b69e-e8bb133ec73e" containerName="istio-proxy" Nov 03 20:41:25.802232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.802191 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eef084f-fa41-48aa-becf-55d52edd3050" containerName="istio-proxy" Nov 03 20:41:25.806025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.806004 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:25.811722 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.811697 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dce009\"" Nov 03 20:41:25.811847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.811710 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"agc-cred-b4ad07a0bd8a\"" Nov 03 20:41:25.813556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.813529 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"payload-processing-external-model-e2e--4cfe9bda-dockercfg-p7zkl\"" Nov 03 20:41:25.818079 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.817833 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:25.818968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.818940 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw"] Nov 03 20:41:25.856388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.856355 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:25.856637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.856432 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:25.856720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.856702 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:25.856850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.856831 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npt4l\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-kube-api-access-npt4l\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:25.940517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.938786 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"kube-root-ca.crt\"" Nov 03 20:41:25.940517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.938917 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"routing-overlay\"" Nov 03 20:41:25.940517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.938985 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"openshift-service-ca.crt\"" Nov 03 20:41:25.980469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:25.959527 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.017487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.017438 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npt4l\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-kube-api-access-npt4l\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.017818 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.017794 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.018004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.017989 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.019068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.019039 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.019230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.019173 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.049469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.045836 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.078468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.078351 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npt4l\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-kube-api-access-npt4l\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.130713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.129042 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.461264 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:41:26.461214 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce91d954_23cd_40c5_a970_0fc15faafe15.slice/crio-b82aced05a75c1a8eef6b7887a1e44851dcfb8f7020f57ed658afb726a3a7da5 WatchSource:0}: Error finding container b82aced05a75c1a8eef6b7887a1e44851dcfb8f7020f57ed658afb726a3a7da5: Status 404 returned error can't find the container with id b82aced05a75c1a8eef6b7887a1e44851dcfb8f7020f57ed658afb726a3a7da5 Nov 03 20:41:26.466280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.466168 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw"] Nov 03 20:41:26.712713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.712668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:26.861896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.861859 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:26.872731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.872696 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" event={"ID":"ce91d954-23cd-40c5-a970-0fc15faafe15","Type":"ContainerStarted","Data":"8cc079c5e53ff8a2128879124284ad9a798cb332548539d69aa9d940a8ed4d79"} Nov 03 20:41:26.874330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.874303 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" event={"ID":"ce91d954-23cd-40c5-a970-0fc15faafe15","Type":"ContainerStarted","Data":"b82aced05a75c1a8eef6b7887a1e44851dcfb8f7020f57ed658afb726a3a7da5"} Nov 03 20:41:26.874755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.874737 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:26.950468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:26.948038 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" podStartSLOduration=1.948012759 podStartE2EDuration="1.948012759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:26.900069335 +0000 UTC m=+2390.350175381" watchObservedRunningTime="2026-11-03 20:41:26.948012759 +0000 UTC m=+2390.398118805" Nov 03 20:41:27.188222 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:27.188146 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:27.349188 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:27.349099 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:27.492418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:27.492325 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:28.313904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:28.313865 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:28.997144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:28.997085 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:29.708976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:29.708945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:30.395805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:30.395768 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:31.120530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:31.120493 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:31.907164 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:31.907130 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:32.635250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:32.635206 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:33.503490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:33.503389 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:33.905544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:33.905474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:34.034225 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:34.034182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:34.168319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:34.168236 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:34.341531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:34.339617 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:34.993870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:34.993815 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:35.301016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:35.300983 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:35.822713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:35.822670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:36.511483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:36.511427 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:37.330959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:37.330923 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:37.368899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:37.368857 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:41:37.376562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:37.376533 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:41:37.380293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:37.380268 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:41:37.383498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:37.383437 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:41:37.660161 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:37.660065 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:37.863259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:37.863217 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:38.404049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:38.404008 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:38.713069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:38.712984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:38.851995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:38.851957 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:38.940467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:38.937633 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:41:39.265719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:39.265683 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:39.411994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:39.411956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:39.865146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:39.865103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:40.501069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:40.501032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:40.850341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:40.850303 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:40.987179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:40.987131 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:41.118399 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:41.118319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:41.309584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:41.309545 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:41.447639 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:41.447519 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:41.603089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:41.603050 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:42.128492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:42.128424 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:42.260663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:42.260631 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:43.163088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:43.163046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:43.596510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:43.594921 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:44.128677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:44.128206 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:45.203566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:45.203520 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:46.002807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:46.002758 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:46.204948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:46.204919 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:46.355908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:46.355866 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:46.506502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:46.506393 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:46.760422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:46.757892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:47.496047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:47.496007 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:47.588109 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:41:47.588071 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:41:48.152114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:48.151996 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:48.834021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:48.833978 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:49.226831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:49.226750 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:49.359961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:49.359803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:49.634698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:49.634658 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:50.390137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:50.390098 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:51.201209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:51.201172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:51.913230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:51.913199 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:52.601833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:52.601797 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:52.804163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:52.804122 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:52.936655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:52.936567 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:53.301316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:53.301281 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:53.740169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:53.740082 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:53.871478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:53.868987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:54.374769 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:54.374730 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:54.523831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:54.523794 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:55.375985 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:55.375946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:55.516926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:55.516879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:56.918813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:56.918776 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:57.722395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:57.722339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:58.045085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:58.045051 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:58.407368 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:58.407279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:58.895383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:58.895351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:41:59.430623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:41:59.430591 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:01.476489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:01.476386 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:02.629637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:02.629599 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:03.736646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:03.736608 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:04.614666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:04.614632 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:05.407217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:05.407182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:06.277859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:06.277820 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:07.091731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:07.091693 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:07.813663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:07.813626 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:08.607632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:08.607586 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:09.307688 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:09.307647 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:10.017395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:10.017333 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:10.700401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:10.700358 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:11.423943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:11.423881 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:12.127681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:12.127436 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:12.882977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:12.882938 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:13.541180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:13.541142 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:14.226529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:14.226492 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:14.975188 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:14.975156 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:15.665752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:15.665689 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:16.371543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:16.371288 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:17.128536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:17.127732 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:17.535429 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:42:17.535391 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:42:17.957786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:17.957608 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:18.696725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:18.696694 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:19.360112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:19.360044 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:20.219211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:20.219167 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:21.013546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:21.013323 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:21.752545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:21.752490 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:22.405093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:22.405054 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:23.088680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:23.088649 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:23.814799 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:23.814767 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:24.503134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:24.503096 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:25.207577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:25.207539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:26.014387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:26.014354 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:26.794730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:26.794698 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:27.478411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:27.478375 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:28.191031 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:28.190753 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:28.825294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:28.825254 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:29.502541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:29.502497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:30.194580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:30.194536 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:30.862882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:30.862843 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:31.611224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:31.611191 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:32.321479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:32.318261 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:32.984844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:32.984806 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:33.723779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:33.723748 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:34.410001 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:34.409954 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:35.100053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:35.100019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:35.867502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:35.867464 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:36.843470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:36.843012 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:37.914979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:37.914943 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:38.796477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:38.795371 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:39.855655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:39.855614 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:40.584595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:40.584560 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:40.723942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:40.723901 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:41.485640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:41.485605 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:42.166960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:42.166923 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:42.825746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:42.825699 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:43.514510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:43.514474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:44.196951 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:44.196906 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:44.917978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:44.917939 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:45.657305 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:45.657262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:46.408118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:46.408083 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:46.537733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:46.537685 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:47.109262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:47.109230 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:47.536285 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:42:47.536247 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:42:47.993292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:47.993203 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:48.871020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:48.870961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:49.555844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:49.555803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:50.420483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:50.418117 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:51.200634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:51.200602 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:51.896431 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:51.896393 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:52.691392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:52.691355 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:53.490493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:53.490425 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:54.130694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:54.130654 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:54.813290 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:54.813245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:55.492799 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:55.492769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:56.211507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:56.211470 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:57.000517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:57.000474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:57.657758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:57.657707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:58.104382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:58.104327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:58.365042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:58.364950 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:58.947762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:58.947728 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:59.094372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:59.094336 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:42:59.907498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:42:59.907460 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:00.593531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:00.593485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:01.436884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:01.436845 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:02.107225 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:02.107192 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:02.879993 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:02.879959 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:03.719848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:03.719803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:04.719656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:04.717183 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:05.360180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:05.360150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:06.023209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:06.023173 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:06.721160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:06.721124 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:07.406566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:07.406524 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:08.193982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:08.193949 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:09.014542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:09.014485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:09.696963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:09.696930 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:10.409621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:10.409586 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:11.199619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:11.199580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:11.909174 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:11.909139 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:12.627659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:12.627621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:13.637475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:13.626659 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:14.834957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:14.834922 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:15.763066 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:15.763037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:17.043171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:17.043127 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:17.551569 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:43:17.551525 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:43:18.032834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:18.032793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:18.856324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:18.856289 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:18.989990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:18.989951 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:19.489779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:19.489742 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:19.619320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:19.619275 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:19.812155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:19.812106 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:19.966557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:19.963023 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:20.352537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:20.352498 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:20.498814 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:20.498753 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:21.481394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:21.481361 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:22.213125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:22.213068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:22.963201 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:22.963162 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:23.843231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:23.842826 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:24.510192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:24.510160 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:25.185833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:25.185801 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:25.979686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:25.978763 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:26.703353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:26.703317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:27.470467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:27.469362 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:28.204561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:28.204520 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:28.909947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:28.909907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:29.586216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:29.586137 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:30.368479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:30.365940 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:31.275945 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:31.275900 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:31.402605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:31.402555 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:31.531638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:31.531550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:31.998788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:31.998699 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:32.446649 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:32.446609 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:33.306817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:33.306779 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:34.103803 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:34.103721 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:34.693363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:34.693323 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:34.825280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:34.825239 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:35.301472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:35.298103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:36.119093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:36.119052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:36.587851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:36.587816 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:36.801525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:36.801484 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:37.064894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:37.064854 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:37.207526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:37.207485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:37.828011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:37.827956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:38.510637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:38.510605 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:39.158592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:39.158550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:39.323132 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:39.323100 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:39.453007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:39.452899 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:40.188186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:40.188147 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:40.413632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:40.413595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:40.547285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:40.547251 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:40.893707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:40.893609 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:41.061081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:41.061047 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:41.893870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:41.893836 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:42.149756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:42.149673 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:42.293025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:42.292047 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:42.826134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:42.826098 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:43.525128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:43.525089 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:44.334304 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:44.334271 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:45.144963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:45.144924 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:45.753347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:45.753312 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:45.878926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:45.878885 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:46.005978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:46.005945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:46.617984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:46.617939 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:46.764472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:46.762060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:47.210262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.210231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:47.229165 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.229132 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb"] Nov 03 20:43:47.234680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.234655 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.239969 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.239943 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 20:43:47.240668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.240643 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb"] Nov 03 20:43:47.240913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.240888 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 20:43:47.241039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.240965 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 20:43:47.241039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.241035 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 20:43:47.260671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.249500 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9e8fb75621d0\"" Nov 03 20:43:47.260671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.249622 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vdt4z\"" Nov 03 20:43:47.410471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.404843 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-routing-overlay-volume\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.410471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.409576 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-provider-credentials\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.410471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.410070 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshbq\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-kube-api-access-lshbq\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.410471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.410247 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-plugins-config-volume\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.525196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.514501 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshbq\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-kube-api-access-lshbq\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.525196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.514570 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-plugins-config-volume\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.525196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.514795 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-routing-overlay-volume\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.525196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.514837 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-provider-credentials\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.525196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.516607 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-plugins-config-volume\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.525196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.517117 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-routing-overlay-volume\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.553464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.547554 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-provider-credentials\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.563490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.561535 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshbq\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-kube-api-access-lshbq\") pod \"payload-processing-external-model-6cfb9f578f-8m4nb\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.725721 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:43:47.725678 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:43:47.854133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.854052 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:47.929499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.926635 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:47.961296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:47.961260 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb"] Nov 03 20:43:48.054241 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:48.054206 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:48.236614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:48.234259 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb"] Nov 03 20:43:48.288488 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:43:48.286087 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72f329e_d42d_43a4_9de6_374ae3781af6.slice/crio-4905a384046ebd864a24867739175383f15e22851661e25d419b746706d745e2 WatchSource:0}: Error finding container 4905a384046ebd864a24867739175383f15e22851661e25d419b746706d745e2: Status 404 returned error can't find the container with id 4905a384046ebd864a24867739175383f15e22851661e25d419b746706d745e2 Nov 03 20:43:49.049858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:49.039956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:49.369344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:49.369236 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" event={"ID":"b72f329e-d42d-43a4-9de6-374ae3781af6","Type":"ContainerStarted","Data":"0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da"} Nov 03 20:43:49.369344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:49.369345 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:49.369582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:49.369363 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" event={"ID":"b72f329e-d42d-43a4-9de6-374ae3781af6","Type":"ContainerStarted","Data":"4905a384046ebd864a24867739175383f15e22851661e25d419b746706d745e2"} Nov 03 20:43:49.391238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:49.391182 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" podStartSLOduration=2.391167892 podStartE2EDuration="2.391167892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:49.388964348 +0000 UTC m=+2532.839070395" watchObservedRunningTime="2026-11-03 20:43:49.391167892 +0000 UTC m=+2532.841273937" Nov 03 20:43:49.809073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:49.809024 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:50.540942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:50.540907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:51.102466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:51.102417 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:51.308254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:51.308214 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:51.586435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:51.586396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:51.733546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:51.733483 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:52.286272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:52.286242 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:53.113669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:53.113635 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:53.905947 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:53.905909 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:54.369630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.369553 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" podUID="b72f329e-d42d-43a4-9de6-374ae3781af6" containerName="payload-processing" containerID="cri-o://0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da" gracePeriod=25 Nov 03 20:43:54.670668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.670641 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:54.701606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.701574 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-kube-api-access-lshbq\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-kube-api-access-lshbq\") pod \"b72f329e-d42d-43a4-9de6-374ae3781af6\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " Nov 03 20:43:54.701782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.701764 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-routing-overlay-volume\") pod \"b72f329e-d42d-43a4-9de6-374ae3781af6\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " Nov 03 20:43:54.701876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.701815 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-provider-credentials\") pod \"b72f329e-d42d-43a4-9de6-374ae3781af6\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " Nov 03 20:43:54.702024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.702004 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-plugins-config-volume\") pod \"b72f329e-d42d-43a4-9de6-374ae3781af6\" (UID: \"b72f329e-d42d-43a4-9de6-374ae3781af6\") " Nov 03 20:43:54.702169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.702141 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-routing-overlay-volume" pod "b72f329e-d42d-43a4-9de6-374ae3781af6" (UID: "b72f329e-d42d-43a4-9de6-374ae3781af6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:54.702690 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.702661 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-plugins-config-volume" pod "b72f329e-d42d-43a4-9de6-374ae3781af6" (UID: "b72f329e-d42d-43a4-9de6-374ae3781af6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:43:54.703607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.703582 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:54.703714 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.703612 4711 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b72f329e-d42d-43a4-9de6-374ae3781af6-routing-overlay-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:54.704750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.704718 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-kube-api-access-lshbq" pod "b72f329e-d42d-43a4-9de6-374ae3781af6" (UID: "b72f329e-d42d-43a4-9de6-374ae3781af6"). InnerVolumeSpecName "kube-api-access-lshbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:54.704857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.704819 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-provider-credentials" pod "b72f329e-d42d-43a4-9de6-374ae3781af6" (UID: "b72f329e-d42d-43a4-9de6-374ae3781af6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:43:54.805050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.805019 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lshbq\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-kube-api-access-lshbq\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:54.805050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.805046 4711 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b72f329e-d42d-43a4-9de6-374ae3781af6-provider-credentials\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:43:54.914061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:54.913981 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:55.387527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.373374 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:55.437470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.434309 4711 generic.go:356] "Generic (PLEG): container finished" podID="b72f329e-d42d-43a4-9de6-374ae3781af6" containerID="0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da" exitCode=0 Nov 03 20:43:55.437470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.434460 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" event={"ID":"b72f329e-d42d-43a4-9de6-374ae3781af6","Type":"ContainerDied","Data":"0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da"} Nov 03 20:43:55.437470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.434516 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" event={"ID":"b72f329e-d42d-43a4-9de6-374ae3781af6","Type":"ContainerDied","Data":"4905a384046ebd864a24867739175383f15e22851661e25d419b746706d745e2"} Nov 03 20:43:55.437470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.434548 4711 scope.go:122] "RemoveContainer" containerID="0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da" Nov 03 20:43:55.437470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.434705 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb" Nov 03 20:43:55.479609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.479584 4711 scope.go:122] "RemoveContainer" containerID="0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da" Nov 03 20:43:55.480069 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:43:55.480049 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da\": container with ID starting with 0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da not found: ID does not exist" containerID="0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da" Nov 03 20:43:55.480168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.480082 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da"} err="failed to get container status \"0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da\": rpc error: code = NotFound desc = could not find container \"0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da\": container with ID starting with 0829461fe2cefb0e448fc6d44fb0f451e8e8667ab3d6788d3b502bc6495f07da not found: ID does not exist" Nov 03 20:43:55.628795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.611084 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb"] Nov 03 20:43:55.632538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.630428 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:55.638759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:55.638683 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6cfb9f578f-8m4nb"] Nov 03 20:43:56.099470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:56.099088 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:56.754423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:56.754377 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72f329e-d42d-43a4-9de6-374ae3781af6" path="/var/lib/kubelet/pods/b72f329e-d42d-43a4-9de6-374ae3781af6/volumes" Nov 03 20:43:56.788516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:56.788486 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:56.961338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:56.961185 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:56.988541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:56.974499 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq"] Nov 03 20:43:57.215951 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.211688 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b72f329e-d42d-43a4-9de6-374ae3781af6" containerName="payload-processing" Nov 03 20:43:57.215951 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.211743 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72f329e-d42d-43a4-9de6-374ae3781af6" containerName="payload-processing" Nov 03 20:43:57.215951 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.211961 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72f329e-d42d-43a4-9de6-374ae3781af6" containerName="payload-processing" Nov 03 20:43:57.394809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.394772 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:43:57.408029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.407997 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8c6934\"" Nov 03 20:43:57.414513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.408848 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8c6934-dockercfg-f6wtr\"" Nov 03 20:43:57.443861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.434980 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-947bh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:43:57.443861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.435051 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:43:57.551881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.551841 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a223b44-25c7-49bf-96d2-e166e1d59604-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq\" (UID: \"4a223b44-25c7-49bf-96d2-e166e1d59604\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:43:57.552100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.552007 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85b72\" (UniqueName: \"kubernetes.io/projected/4a223b44-25c7-49bf-96d2-e166e1d59604-kube-api-access-85b72\") pod \"payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq\" (UID: \"4a223b44-25c7-49bf-96d2-e166e1d59604\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:43:57.554294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.446377 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq"] Nov 03 20:43:57.654272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.653840 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a223b44-25c7-49bf-96d2-e166e1d59604-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq\" (UID: \"4a223b44-25c7-49bf-96d2-e166e1d59604\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:43:57.654272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.653886 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85b72\" (UniqueName: \"kubernetes.io/projected/4a223b44-25c7-49bf-96d2-e166e1d59604-kube-api-access-85b72\") pod \"payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq\" (UID: \"4a223b44-25c7-49bf-96d2-e166e1d59604\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:43:57.654600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.654575 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a223b44-25c7-49bf-96d2-e166e1d59604-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq\" (UID: \"4a223b44-25c7-49bf-96d2-e166e1d59604\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:43:57.692430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.692394 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85b72\" (UniqueName: \"kubernetes.io/projected/4a223b44-25c7-49bf-96d2-e166e1d59604-kube-api-access-85b72\") pod \"payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq\" (UID: \"4a223b44-25c7-49bf-96d2-e166e1d59604\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:43:57.731184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.731116 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:43:57.775327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:57.755058 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:58.143101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:58.143072 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq"] Nov 03 20:43:58.144152 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:43:58.144118 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a223b44_25c7_49bf_96d2_e166e1d59604.slice/crio-3d4326047fdd551c25c6158727e5f66be66c303983b68ac5a43ebc1ff04756d0 WatchSource:0}: Error finding container 3d4326047fdd551c25c6158727e5f66be66c303983b68ac5a43ebc1ff04756d0: Status 404 returned error can't find the container with id 3d4326047fdd551c25c6158727e5f66be66c303983b68ac5a43ebc1ff04756d0 Nov 03 20:43:58.283898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:58.283861 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:58.622312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:58.622266 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" event={"ID":"4a223b44-25c7-49bf-96d2-e166e1d59604","Type":"ContainerStarted","Data":"d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5"} Nov 03 20:43:58.622536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:58.622332 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" event={"ID":"4a223b44-25c7-49bf-96d2-e166e1d59604","Type":"ContainerStarted","Data":"3d4326047fdd551c25c6158727e5f66be66c303983b68ac5a43ebc1ff04756d0"} Nov 03 20:43:58.622536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:58.622375 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:43:58.644759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:58.644707 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" podStartSLOduration=2.644692704 podStartE2EDuration="2.644692704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:58.640869529 +0000 UTC m=+2542.090975588" watchObservedRunningTime="2026-11-03 20:43:58.644692704 +0000 UTC m=+2542.094798770" Nov 03 20:43:58.885529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:58.884436 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:43:59.703553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:43:59.703523 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:00.555429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:00.555395 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:01.352862 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:01.352830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:02.110578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:02.110539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:02.237135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:02.237100 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:02.364780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:02.364678 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:03.005235 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:03.005203 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:03.732594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:03.732557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:04.558851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:04.558812 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:05.297849 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:05.297808 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:06.085618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:06.085557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:06.818090 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:06.818054 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:07.562470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:07.557652 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:08.201259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:08.201227 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:08.907739 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:08.907693 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:09.632344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:09.632315 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:44:09.793157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:09.793122 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:10.552321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:10.552279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:11.291788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:11.291748 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:11.988229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:11.988195 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:12.717572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:12.717540 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:13.407480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:13.396961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:14.120967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:14.120930 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:14.849276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:14.849173 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:15.493224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:15.493184 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:15.635053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:15.635010 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:16.498589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:16.498550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:17.299772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:17.299740 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:17.534582 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:44:17.534544 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:44:18.083538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:18.083502 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:18.814281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:18.814251 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:19.544479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:19.542201 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:20.196635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:20.196602 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:21.023096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:21.023062 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:21.712113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:21.712079 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:22.678149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:22.678112 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:23.473267 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:23.473118 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:24.136642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:24.136600 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:24.899583 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:24.899542 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:25.612221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:25.612188 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:26.294848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:26.294808 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:26.654569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:26.654439 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:27.027880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:27.027841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:27.166949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:27.166912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:27.551150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:27.551103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:27.683280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:27.683248 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:28.130602 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:28.130556 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:28.797168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:28.797125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:29.579830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:29.579758 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:30.356701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:30.356663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:31.197647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:31.197604 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:31.717350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:31.717313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:31.852662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:31.852627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:32.168459 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:32.168410 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:32.883025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:32.882986 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:33.098076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.098034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:33.144793 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.144694 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn"] Nov 03 20:44:33.158326 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.157526 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:33.160470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.160429 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn"] Nov 03 20:44:33.228848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.228818 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c94fdf\"" Nov 03 20:44:33.229029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.228954 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c94fdf-dockercfg-9xzvn\"" Nov 03 20:44:33.230786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.230757 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:33.253548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.253518 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqtb\" (UniqueName: \"kubernetes.io/projected/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-kube-api-access-mxqtb\") pod \"payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn\" (UID: \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:33.253728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.253707 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn\" (UID: \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:33.396526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.394843 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqtb\" (UniqueName: \"kubernetes.io/projected/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-kube-api-access-mxqtb\") pod \"payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn\" (UID: \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:33.396526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.395123 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn\" (UID: \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:33.405467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.397879 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn\" (UID: \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:33.462468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.462049 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqtb\" (UniqueName: \"kubernetes.io/projected/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-kube-api-access-mxqtb\") pod \"payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn\" (UID: \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:33.489471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.478741 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:33.940902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.936825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:33.974030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:33.973992 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn"] Nov 03 20:44:34.006974 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:44:34.005725 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95e60b8_b0ce_400e_bdd7_cd9c4265943a.slice/crio-ad050739bb2231b0cbc2a252b271652684bcbdf78db7c91f7e24c0a2bb2e3a1e WatchSource:0}: Error finding container ad050739bb2231b0cbc2a252b271652684bcbdf78db7c91f7e24c0a2bb2e3a1e: Status 404 returned error can't find the container with id ad050739bb2231b0cbc2a252b271652684bcbdf78db7c91f7e24c0a2bb2e3a1e Nov 03 20:44:34.055377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:34.055254 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" event={"ID":"f95e60b8-b0ce-400e-bdd7-cd9c4265943a","Type":"ContainerStarted","Data":"ad050739bb2231b0cbc2a252b271652684bcbdf78db7c91f7e24c0a2bb2e3a1e"} Nov 03 20:44:35.085585 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:35.085538 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" event={"ID":"f95e60b8-b0ce-400e-bdd7-cd9c4265943a","Type":"ContainerStarted","Data":"d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b"} Nov 03 20:44:35.088163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:35.087067 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:35.106410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:35.106357 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" podStartSLOduration=2.106343981 podStartE2EDuration="2.106343981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:35.106211344 +0000 UTC m=+2578.556317389" watchObservedRunningTime="2026-11-03 20:44:35.106343981 +0000 UTC m=+2578.556450025" Nov 03 20:44:35.159861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:35.159818 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:35.941498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:35.941459 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:36.705321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:36.705289 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:37.697472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:37.685724 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:38.329273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:38.329227 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:39.099712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:39.099669 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:39.798753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:39.798716 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:40.624462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:40.622003 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:41.140195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:41.140155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:41.407157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:41.407056 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:41.592151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:41.592116 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:41.969823 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:41.969787 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n"] Nov 03 20:44:41.976249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:41.976221 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:41.986729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:41.986699 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f5afd7-openshift-default-dockercfg-vz568\"" Nov 03 20:44:41.987556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:41.987529 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n"] Nov 03 20:44:42.053209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.053176 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:42.135906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.135868 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-socket\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.136083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.135990 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istiod-ca-cert\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.136083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.136060 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-data\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.136194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.136102 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-podinfo\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.136194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.136156 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbwf\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-kube-api-access-5dbwf\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.136302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.136283 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-token\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.136356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.136340 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-credential-socket\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.136483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.136435 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-certs\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.136537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.136519 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-envoy\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.180335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.180300 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:42.313351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.313306 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-certs\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.317354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.317328 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-envoy\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.317749 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.317728 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-socket\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.317962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.317946 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istiod-ca-cert\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.318113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.318098 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-data\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.318253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.318235 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-podinfo\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.318423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.318409 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbwf\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-kube-api-access-5dbwf\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.318616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.318598 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-token\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.318785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.318770 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-credential-socket\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.319395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.319374 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-credential-socket\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.331540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.320197 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istiod-ca-cert\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.331700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.317212 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-certs\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.331832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.325214 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-socket\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.333894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.333872 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-data\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.359102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.356965 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:42.420465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.419083 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-podinfo\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.436507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.430030 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-envoy\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.436507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.430687 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbwf\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-kube-api-access-5dbwf\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.460465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.441935 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-token\") pod \"e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.627684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.627603 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:42.970533 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:44:42.970501 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ed0c60_5fec_4a41_912b_f5f2e9ff1516.slice/crio-34a8a4ba475d71fa963542a0e3ff1ca3f451ee671237b0b21eed9e26d34c93bf WatchSource:0}: Error finding container 34a8a4ba475d71fa963542a0e3ff1ca3f451ee671237b0b21eed9e26d34c93bf: Status 404 returned error can't find the container with id 34a8a4ba475d71fa963542a0e3ff1ca3f451ee671237b0b21eed9e26d34c93bf Nov 03 20:44:42.970800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.970779 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n"] Nov 03 20:44:42.973413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.973375 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:44:42.973540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.973500 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:44:42.973612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:42.973550 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:44:43.119112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:43.119075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:43.223065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:43.222922 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" event={"ID":"84ed0c60-5fec-4a41-912b-f5f2e9ff1516","Type":"ContainerStarted","Data":"560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115"} Nov 03 20:44:43.223065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:43.222996 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" event={"ID":"84ed0c60-5fec-4a41-912b-f5f2e9ff1516","Type":"ContainerStarted","Data":"34a8a4ba475d71fa963542a0e3ff1ca3f451ee671237b0b21eed9e26d34c93bf"} Nov 03 20:44:43.253182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:43.253121 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podStartSLOduration=2.25310278 podStartE2EDuration="2.25310278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:43.248281474 +0000 UTC m=+2586.698387522" watchObservedRunningTime="2026-11-03 20:44:43.25310278 +0000 UTC m=+2586.703208827" Nov 03 20:44:43.629395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:43.629352 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:43.908865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:43.908777 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:44.634277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:44.634250 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:44.758701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:44.758661 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:45.240567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:45.240430 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:45.252342 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:45.241415 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:44:45.478897 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:45.478849 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:45.613471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:45.610719 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:46.599193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:46.599155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:46.738128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:46.738090 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:46.931531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:46.919233 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:47.111578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:47.110186 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:44:47.533745 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:44:47.533711 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:44:47.727466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:47.723517 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:48.337620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:48.337589 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:49.023335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:49.023301 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:49.697321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:49.697288 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:50.398672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:50.398626 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:51.184037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:51.184003 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:51.900525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:51.900491 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:52.547029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.546993 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q"] Nov 03 20:44:52.558628 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.558597 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q"] Nov 03 20:44:52.558975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.558933 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.563863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.563842 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c94fdf-serving-cert\"" Nov 03 20:44:52.584647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.584617 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.584832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.584786 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-metrics-certs\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.585074 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.585053 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-maas-api-tls\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.585152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.585089 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd72p\" (UniqueName: \"kubernetes.io/projected/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-kube-api-access-qd72p\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.617465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.617421 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:52.720465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.717303 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-metrics-certs\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.720465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.717581 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-maas-api-tls\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.720465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.717619 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd72p\" (UniqueName: \"kubernetes.io/projected/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-kube-api-access-qd72p\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.720465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.717885 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.721946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.721922 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.778520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.770527 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd72p\" (UniqueName: \"kubernetes.io/projected/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-kube-api-access-qd72p\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.778520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.771237 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-maas-api-tls\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.778520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.772626 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:52.789204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.787243 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-metrics-certs\") pod \"maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:52.882934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:52.882847 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:53.220520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:53.217024 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q"] Nov 03 20:44:53.259469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:53.249141 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:44:53.339892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:53.339860 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" event={"ID":"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4","Type":"ContainerStarted","Data":"d8dae52b8aff6785113b7b8a06ed232152a0e3d78b9128453f12c8dd286a9551"} Nov 03 20:44:53.521108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:53.521073 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:54.307691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:54.307659 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:54.369625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:54.369588 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" event={"ID":"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4","Type":"ContainerStarted","Data":"58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62"} Nov 03 20:44:54.369938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:54.369918 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:44:54.392920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:54.392868 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" podStartSLOduration=1.814862169 podStartE2EDuration="2.392849547s" totalImagesPullingTime="577.987378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:54.389968232 +0000 UTC m=+2597.840074278" watchObservedRunningTime="2026-11-03 20:44:54.392849547 +0000 UTC m=+2597.842955593" Nov 03 20:44:55.106953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:55.106912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:55.796673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:55.796632 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:56.501180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:56.501112 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:57.221498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:57.219734 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:58.004115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:58.004071 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:58.343425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:58.343388 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:58.697019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:58.696933 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:59.153807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:59.153766 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:44:59.418086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:44:59.418004 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:00.087314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.087272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:00.202595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.202561 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg"] Nov 03 20:45:00.209299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.209254 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:00.211127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.211098 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl"] Nov 03 20:45:00.250220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.244848 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg"] Nov 03 20:45:00.250220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.244895 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq"] Nov 03 20:45:00.250220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.245276 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:00.253740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.253703 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z"] Nov 03 20:45:00.254109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.253817 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:00.262878 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.262464 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq"] Nov 03 20:45:00.262878 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.262562 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:00.282141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.278611 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf"] Nov 03 20:45:00.283917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.283893 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:00.284487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.284471 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl"] Nov 03 20:45:00.291573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.291553 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z"] Nov 03 20:45:00.298949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.298925 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk"] Nov 03 20:45:00.305233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.305198 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6aeae1-eeef-47ee-8798-438f7577a6c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-wmz8z\" (UID: \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:00.305599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.305578 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhw9\" (UniqueName: \"kubernetes.io/projected/e41bb147-1f27-47bb-a3be-4c3538170edd-kube-api-access-bnhw9\") pod \"maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq\" (UID: \"e41bb147-1f27-47bb-a3be-4c3538170edd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:00.305820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.305802 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlpws\" (UniqueName: \"kubernetes.io/projected/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-kube-api-access-wlpws\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg\" (UID: \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:00.305973 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.305961 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41bb147-1f27-47bb-a3be-4c3538170edd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq\" (UID: \"e41bb147-1f27-47bb-a3be-4c3538170edd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:00.306101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.306091 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg\" (UID: \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:00.306202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.306192 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srh7\" (UniqueName: \"kubernetes.io/projected/fc6aeae1-eeef-47ee-8798-438f7577a6c1-kube-api-access-2srh7\") pod \"maas-api-key-cleanup-29895645-wmz8z\" (UID: \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:00.306353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.306343 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2423b82-2e79-4702-9a43-4e246f7eab55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf\" (UID: \"b2423b82-2e79-4702-9a43-4e246f7eab55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:00.306483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.306469 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls8nc\" (UniqueName: \"kubernetes.io/projected/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-kube-api-access-ls8nc\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl\" (UID: \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:00.307263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.307248 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fsn\" (UniqueName: \"kubernetes.io/projected/b2423b82-2e79-4702-9a43-4e246f7eab55-kube-api-access-p6fsn\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf\" (UID: \"b2423b82-2e79-4702-9a43-4e246f7eab55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:00.307740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.307727 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 20:45:00.307874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.307860 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl\" (UID: \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:00.308229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.308213 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf"] Nov 03 20:45:00.308328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.308314 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:00.312426 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.312412 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk"] Nov 03 20:45:00.322659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.322639 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf"] Nov 03 20:45:00.327803 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.327778 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:00.337084 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.337063 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf"] Nov 03 20:45:00.382149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.382122 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:45:00.409856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.409616 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl\" (UID: \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:00.409856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.409674 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk\" (UID: \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:00.409856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.409781 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd49149-1509-4696-8418-bfe975d02584-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf\" (UID: \"9bd49149-1509-4696-8418-bfe975d02584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:00.410321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.409886 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6aeae1-eeef-47ee-8798-438f7577a6c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-wmz8z\" (UID: \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:00.410321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.410038 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhw9\" (UniqueName: \"kubernetes.io/projected/e41bb147-1f27-47bb-a3be-4c3538170edd-kube-api-access-bnhw9\") pod \"maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq\" (UID: \"e41bb147-1f27-47bb-a3be-4c3538170edd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:00.410321 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.410065 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxskp\" (UniqueName: \"kubernetes.io/projected/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-kube-api-access-mxskp\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk\" (UID: \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:00.410433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.410396 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlpws\" (UniqueName: \"kubernetes.io/projected/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-kube-api-access-wlpws\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg\" (UID: \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:00.410506 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.410428 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41bb147-1f27-47bb-a3be-4c3538170edd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq\" (UID: \"e41bb147-1f27-47bb-a3be-4c3538170edd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:00.410567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.410541 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg\" (UID: \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:00.410658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.410640 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srh7\" (UniqueName: \"kubernetes.io/projected/fc6aeae1-eeef-47ee-8798-438f7577a6c1-kube-api-access-2srh7\") pod \"maas-api-key-cleanup-29895645-wmz8z\" (UID: \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:00.412124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.410946 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2423b82-2e79-4702-9a43-4e246f7eab55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf\" (UID: \"b2423b82-2e79-4702-9a43-4e246f7eab55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:00.412124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.410988 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rj6\" (UniqueName: \"kubernetes.io/projected/9bd49149-1509-4696-8418-bfe975d02584-kube-api-access-68rj6\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf\" (UID: \"9bd49149-1509-4696-8418-bfe975d02584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:00.412124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.411024 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls8nc\" (UniqueName: \"kubernetes.io/projected/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-kube-api-access-ls8nc\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl\" (UID: \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:00.412124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.411096 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fsn\" (UniqueName: \"kubernetes.io/projected/b2423b82-2e79-4702-9a43-4e246f7eab55-kube-api-access-p6fsn\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf\" (UID: \"b2423b82-2e79-4702-9a43-4e246f7eab55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:00.412124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.411257 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl\" (UID: \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:00.412124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.411274 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6aeae1-eeef-47ee-8798-438f7577a6c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-wmz8z\" (UID: \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:00.412124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.411368 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41bb147-1f27-47bb-a3be-4c3538170edd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq\" (UID: \"e41bb147-1f27-47bb-a3be-4c3538170edd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:00.412124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.412048 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2423b82-2e79-4702-9a43-4e246f7eab55-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf\" (UID: \"b2423b82-2e79-4702-9a43-4e246f7eab55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:00.412745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.412189 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg\" (UID: \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:00.423640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.423614 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fsn\" (UniqueName: \"kubernetes.io/projected/b2423b82-2e79-4702-9a43-4e246f7eab55-kube-api-access-p6fsn\") pod \"maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf\" (UID: \"b2423b82-2e79-4702-9a43-4e246f7eab55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:00.424959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.424768 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhw9\" (UniqueName: \"kubernetes.io/projected/e41bb147-1f27-47bb-a3be-4c3538170edd-kube-api-access-bnhw9\") pod \"maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq\" (UID: \"e41bb147-1f27-47bb-a3be-4c3538170edd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:00.425960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.425937 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls8nc\" (UniqueName: \"kubernetes.io/projected/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-kube-api-access-ls8nc\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl\" (UID: \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:00.426467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.426369 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlpws\" (UniqueName: \"kubernetes.io/projected/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-kube-api-access-wlpws\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg\" (UID: \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:00.427516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.427431 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srh7\" (UniqueName: \"kubernetes.io/projected/fc6aeae1-eeef-47ee-8798-438f7577a6c1-kube-api-access-2srh7\") pod \"maas-api-key-cleanup-29895645-wmz8z\" (UID: \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:00.512202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.512165 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxskp\" (UniqueName: \"kubernetes.io/projected/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-kube-api-access-mxskp\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk\" (UID: \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:00.512386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.512270 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rj6\" (UniqueName: \"kubernetes.io/projected/9bd49149-1509-4696-8418-bfe975d02584-kube-api-access-68rj6\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf\" (UID: \"9bd49149-1509-4696-8418-bfe975d02584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:00.512386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.512336 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk\" (UID: \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:00.512489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.512415 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd49149-1509-4696-8418-bfe975d02584-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf\" (UID: \"9bd49149-1509-4696-8418-bfe975d02584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:00.513137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.513110 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd49149-1509-4696-8418-bfe975d02584-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf\" (UID: \"9bd49149-1509-4696-8418-bfe975d02584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:00.513254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.513162 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk\" (UID: \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:00.533497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.528117 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxskp\" (UniqueName: \"kubernetes.io/projected/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-kube-api-access-mxskp\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk\" (UID: \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:00.533497 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.531566 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rj6\" (UniqueName: \"kubernetes.io/projected/9bd49149-1509-4696-8418-bfe975d02584-kube-api-access-68rj6\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf\" (UID: \"9bd49149-1509-4696-8418-bfe975d02584\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:00.565567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.565539 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:00.588411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.588382 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:00.606028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.606000 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:00.612910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.612882 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:00.632806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.632742 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:00.648129 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.648094 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:00.661496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.657681 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:00.863359 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:45:00.861707 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f40772e_a2fa_4b57_ab5a_64a4d571ab88.slice/crio-3f2b3025c7c70a2cf0c582707bd99a5a81cf7f8ffe4ca79665f62b6f6c1d2548 WatchSource:0}: Error finding container 3f2b3025c7c70a2cf0c582707bd99a5a81cf7f8ffe4ca79665f62b6f6c1d2548: Status 404 returned error can't find the container with id 3f2b3025c7c70a2cf0c582707bd99a5a81cf7f8ffe4ca79665f62b6f6c1d2548 Nov 03 20:45:00.865433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.865407 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg"] Nov 03 20:45:00.930586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:00.929887 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:01.102165 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.102136 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl"] Nov 03 20:45:01.106242 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:45:01.106217 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0b88d8_6cbf_4851_a3ec_412c44929ffc.slice/crio-8cdd08a1d217c4004193aba0a2c17170382dd9dfb23d6b2efa676699d92f95e1 WatchSource:0}: Error finding container 8cdd08a1d217c4004193aba0a2c17170382dd9dfb23d6b2efa676699d92f95e1: Status 404 returned error can't find the container with id 8cdd08a1d217c4004193aba0a2c17170382dd9dfb23d6b2efa676699d92f95e1 Nov 03 20:45:01.136354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.136332 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z"] Nov 03 20:45:01.139397 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:45:01.139370 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6aeae1_eeef_47ee_8798_438f7577a6c1.slice/crio-3d205b2d96d7d9b902d9c6b2cbe425950901dbed4405c9f2131a13193f679f26 WatchSource:0}: Error finding container 3d205b2d96d7d9b902d9c6b2cbe425950901dbed4405c9f2131a13193f679f26: Status 404 returned error can't find the container with id 3d205b2d96d7d9b902d9c6b2cbe425950901dbed4405c9f2131a13193f679f26 Nov 03 20:45:01.143382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.143361 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq"] Nov 03 20:45:01.143680 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:45:01.143661 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41bb147_1f27_47bb_a3be_4c3538170edd.slice/crio-0b33b7fb229e4e80801fa437c8d1c635875c92bb69f39e6b4156fdc662e8b771 WatchSource:0}: Error finding container 0b33b7fb229e4e80801fa437c8d1c635875c92bb69f39e6b4156fdc662e8b771: Status 404 returned error can't find the container with id 0b33b7fb229e4e80801fa437c8d1c635875c92bb69f39e6b4156fdc662e8b771 Nov 03 20:45:01.225990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.225962 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk"] Nov 03 20:45:01.260006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.259975 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf"] Nov 03 20:45:01.264139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.264115 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf"] Nov 03 20:45:01.269384 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:45:01.269353 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0c591c_e5ea_4e21_89ec_b8aee7045acb.slice/crio-a07f4d345795a469523d84c0355cfbbdf9d976b99a7cf9e00a27116cb4b93843 WatchSource:0}: Error finding container a07f4d345795a469523d84c0355cfbbdf9d976b99a7cf9e00a27116cb4b93843: Status 404 returned error can't find the container with id a07f4d345795a469523d84c0355cfbbdf9d976b99a7cf9e00a27116cb4b93843 Nov 03 20:45:01.270185 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:45:01.270149 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd49149_1509_4696_8418_bfe975d02584.slice/crio-9e939587fefe2e272cbf9e23e4431fa3cc461644e011ce2523a0130a8232f4fe WatchSource:0}: Error finding container 9e939587fefe2e272cbf9e23e4431fa3cc461644e011ce2523a0130a8232f4fe: Status 404 returned error can't find the container with id 9e939587fefe2e272cbf9e23e4431fa3cc461644e011ce2523a0130a8232f4fe Nov 03 20:45:01.370951 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:45:01.370851 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2423b82_2e79_4702_9a43_4e246f7eab55.slice/crio-7505c7820ccef6253d832d6fc4422a3408df2ff6a3c228452c86a8f29e288461 WatchSource:0}: Error finding container 7505c7820ccef6253d832d6fc4422a3408df2ff6a3c228452c86a8f29e288461: Status 404 returned error can't find the container with id 7505c7820ccef6253d832d6fc4422a3408df2ff6a3c228452c86a8f29e288461 Nov 03 20:45:01.424612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.424585 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" event={"ID":"fd0c591c-e5ea-4e21-89ec-b8aee7045acb","Type":"ContainerStarted","Data":"a07f4d345795a469523d84c0355cfbbdf9d976b99a7cf9e00a27116cb4b93843"} Nov 03 20:45:01.425890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.425860 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" event={"ID":"b2423b82-2e79-4702-9a43-4e246f7eab55","Type":"ContainerStarted","Data":"7505c7820ccef6253d832d6fc4422a3408df2ff6a3c228452c86a8f29e288461"} Nov 03 20:45:01.427154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.427134 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" event={"ID":"9bd49149-1509-4696-8418-bfe975d02584","Type":"ContainerStarted","Data":"9e939587fefe2e272cbf9e23e4431fa3cc461644e011ce2523a0130a8232f4fe"} Nov 03 20:45:01.428411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.428390 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" event={"ID":"4b0b88d8-6cbf-4851-a3ec-412c44929ffc","Type":"ContainerStarted","Data":"8cdd08a1d217c4004193aba0a2c17170382dd9dfb23d6b2efa676699d92f95e1"} Nov 03 20:45:01.429864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.429834 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" event={"ID":"e41bb147-1f27-47bb-a3be-4c3538170edd","Type":"ContainerStarted","Data":"0b33b7fb229e4e80801fa437c8d1c635875c92bb69f39e6b4156fdc662e8b771"} Nov 03 20:45:01.431304 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.431284 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" event={"ID":"8f40772e-a2fa-4b57-ab5a-64a4d571ab88","Type":"ContainerStarted","Data":"3f2b3025c7c70a2cf0c582707bd99a5a81cf7f8ffe4ca79665f62b6f6c1d2548"} Nov 03 20:45:01.432645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.432616 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" event={"ID":"fc6aeae1-eeef-47ee-8798-438f7577a6c1","Type":"ContainerStarted","Data":"3d205b2d96d7d9b902d9c6b2cbe425950901dbed4405c9f2131a13193f679f26"} Nov 03 20:45:01.735470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:01.728934 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:02.443091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.443060 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" event={"ID":"9bd49149-1509-4696-8418-bfe975d02584","Type":"ContainerStarted","Data":"76d07e7ef38d6b59ca2fb02caaeabba2c1d531bfdbf008bf7705ef8c38fe5626"} Nov 03 20:45:02.444907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.444879 4711 generic.go:356] "Generic (PLEG): container finished" podID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerID="5daa11b635cc0c08ae17ef432d104485647edb1efebbf20bebe5e5fbc65fc0e6" exitCode=60 Nov 03 20:45:02.445037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.444962 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" event={"ID":"4b0b88d8-6cbf-4851-a3ec-412c44929ffc","Type":"ContainerDied","Data":"5daa11b635cc0c08ae17ef432d104485647edb1efebbf20bebe5e5fbc65fc0e6"} Nov 03 20:45:02.445261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.445245 4711 scope.go:122] "RemoveContainer" containerID="5daa11b635cc0c08ae17ef432d104485647edb1efebbf20bebe5e5fbc65fc0e6" Nov 03 20:45:02.446766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.446738 4711 generic.go:356] "Generic (PLEG): container finished" podID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerID="67afcc17f4cc22a14bb4685f0a00d1c2821f3e845ea3d4e4ec0e4b9ac04db593" exitCode=60 Nov 03 20:45:02.446865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.446758 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" event={"ID":"e41bb147-1f27-47bb-a3be-4c3538170edd","Type":"ContainerDied","Data":"67afcc17f4cc22a14bb4685f0a00d1c2821f3e845ea3d4e4ec0e4b9ac04db593"} Nov 03 20:45:02.447079 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.447060 4711 scope.go:122] "RemoveContainer" containerID="67afcc17f4cc22a14bb4685f0a00d1c2821f3e845ea3d4e4ec0e4b9ac04db593" Nov 03 20:45:02.448863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.448840 4711 generic.go:356] "Generic (PLEG): container finished" podID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerID="400dd7b7cd31ef5f9b492f4d7b795e8d892b773b6623c4bfec4d62a84a5b0260" exitCode=60 Nov 03 20:45:02.448941 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.448931 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" event={"ID":"8f40772e-a2fa-4b57-ab5a-64a4d571ab88","Type":"ContainerDied","Data":"400dd7b7cd31ef5f9b492f4d7b795e8d892b773b6623c4bfec4d62a84a5b0260"} Nov 03 20:45:02.449175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.449154 4711 scope.go:122] "RemoveContainer" containerID="400dd7b7cd31ef5f9b492f4d7b795e8d892b773b6623c4bfec4d62a84a5b0260" Nov 03 20:45:02.451069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.450983 4711 generic.go:356] "Generic (PLEG): container finished" podID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerID="87e1e12553ba0405a71cd0ee594f1e1d07c946b8270e9fb2e8f6723ac785b675" exitCode=60 Nov 03 20:45:02.451211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.451075 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" event={"ID":"fc6aeae1-eeef-47ee-8798-438f7577a6c1","Type":"ContainerDied","Data":"87e1e12553ba0405a71cd0ee594f1e1d07c946b8270e9fb2e8f6723ac785b675"} Nov 03 20:45:02.451286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.451268 4711 scope.go:122] "RemoveContainer" containerID="87e1e12553ba0405a71cd0ee594f1e1d07c946b8270e9fb2e8f6723ac785b675" Nov 03 20:45:02.452666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.452647 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" event={"ID":"fd0c591c-e5ea-4e21-89ec-b8aee7045acb","Type":"ContainerStarted","Data":"41eb1d04c3e593d9f4d668a61535ed3174804b780bff0863ed0240967aba583d"} Nov 03 20:45:02.454163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.454140 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" event={"ID":"b2423b82-2e79-4702-9a43-4e246f7eab55","Type":"ContainerStarted","Data":"8849207a4546cd3ee3a08e0a5bf7d42469794b551513b4db2d78324c97547029"} Nov 03 20:45:02.466392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.466313 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" podStartSLOduration=2.466297922 podStartE2EDuration="2.466297922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:02.463791047 +0000 UTC m=+2605.913897094" watchObservedRunningTime="2026-11-03 20:45:02.466297922 +0000 UTC m=+2605.916403971" Nov 03 20:45:02.549046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.548818 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:02.616480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.590392 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" podStartSLOduration=2.590371334 podStartE2EDuration="2.590371334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:02.529802555 +0000 UTC m=+2605.979908612" watchObservedRunningTime="2026-11-03 20:45:02.590371334 +0000 UTC m=+2606.040477379" Nov 03 20:45:02.627042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:02.624795 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" podStartSLOduration=2.624775786 podStartE2EDuration="2.624775786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:02.566928428 +0000 UTC m=+2606.017034475" watchObservedRunningTime="2026-11-03 20:45:02.624775786 +0000 UTC m=+2606.074881833" Nov 03 20:45:03.220469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.206314 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:03.463531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.463495 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" event={"ID":"e41bb147-1f27-47bb-a3be-4c3538170edd","Type":"ContainerStarted","Data":"6ab33ec170abb99e410ba1a940a1dc46b2cb2e2794acca9ee1d90ea499b7730e"} Nov 03 20:45:03.465481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.465435 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" event={"ID":"8f40772e-a2fa-4b57-ab5a-64a4d571ab88","Type":"ContainerStarted","Data":"339e40408cad03bc718a46bd53aeeb0bf0747ec17fae264952fd6f17068dc077"} Nov 03 20:45:03.467822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.467793 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" event={"ID":"fc6aeae1-eeef-47ee-8798-438f7577a6c1","Type":"ContainerStarted","Data":"204c7c316137073aefa7adacd43450e13eb67e90c665531679592563f2ea2924"} Nov 03 20:45:03.469613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.469590 4711 generic.go:356] "Generic (PLEG): container finished" podID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerID="41eb1d04c3e593d9f4d668a61535ed3174804b780bff0863ed0240967aba583d" exitCode=60 Nov 03 20:45:03.469742 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.469672 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" event={"ID":"fd0c591c-e5ea-4e21-89ec-b8aee7045acb","Type":"ContainerDied","Data":"41eb1d04c3e593d9f4d668a61535ed3174804b780bff0863ed0240967aba583d"} Nov 03 20:45:03.469857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.469841 4711 scope.go:122] "RemoveContainer" containerID="41eb1d04c3e593d9f4d668a61535ed3174804b780bff0863ed0240967aba583d" Nov 03 20:45:03.471467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.471430 4711 generic.go:356] "Generic (PLEG): container finished" podID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerID="8849207a4546cd3ee3a08e0a5bf7d42469794b551513b4db2d78324c97547029" exitCode=60 Nov 03 20:45:03.471646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.471557 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" event={"ID":"b2423b82-2e79-4702-9a43-4e246f7eab55","Type":"ContainerDied","Data":"8849207a4546cd3ee3a08e0a5bf7d42469794b551513b4db2d78324c97547029"} Nov 03 20:45:03.471715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.471701 4711 scope.go:122] "RemoveContainer" containerID="8849207a4546cd3ee3a08e0a5bf7d42469794b551513b4db2d78324c97547029" Nov 03 20:45:03.473181 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.473160 4711 generic.go:356] "Generic (PLEG): container finished" podID="9bd49149-1509-4696-8418-bfe975d02584" containerID="76d07e7ef38d6b59ca2fb02caaeabba2c1d531bfdbf008bf7705ef8c38fe5626" exitCode=60 Nov 03 20:45:03.473287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.473230 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" event={"ID":"9bd49149-1509-4696-8418-bfe975d02584","Type":"ContainerDied","Data":"76d07e7ef38d6b59ca2fb02caaeabba2c1d531bfdbf008bf7705ef8c38fe5626"} Nov 03 20:45:03.473516 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.473429 4711 scope.go:122] "RemoveContainer" containerID="76d07e7ef38d6b59ca2fb02caaeabba2c1d531bfdbf008bf7705ef8c38fe5626" Nov 03 20:45:03.475562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.475526 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" event={"ID":"4b0b88d8-6cbf-4851-a3ec-412c44929ffc","Type":"ContainerStarted","Data":"95941d65586fc38d90b89010a7c04047d2da2cfa068aeb59c09ccefbe8d07803"} Nov 03 20:45:03.489159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.489109 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" podStartSLOduration=3.489094898 podStartE2EDuration="3.489094898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.4844921 +0000 UTC m=+2606.934598145" watchObservedRunningTime="2026-11-03 20:45:03.489094898 +0000 UTC m=+2606.939200945" Nov 03 20:45:03.563615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.563324 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" podStartSLOduration=3.563307321 podStartE2EDuration="3.563307321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.561633116 +0000 UTC m=+2607.011739163" watchObservedRunningTime="2026-11-03 20:45:03.563307321 +0000 UTC m=+2607.013413368" Nov 03 20:45:03.584680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.584636 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" podStartSLOduration=3.584622713 podStartE2EDuration="3.584622713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.582316262 +0000 UTC m=+2607.032422330" watchObservedRunningTime="2026-11-03 20:45:03.584622713 +0000 UTC m=+2607.034728758" Nov 03 20:45:03.604484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:03.604417 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" podStartSLOduration=3.604404996 podStartE2EDuration="3.604404996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:03.599757535 +0000 UTC m=+2607.049863582" watchObservedRunningTime="2026-11-03 20:45:03.604404996 +0000 UTC m=+2607.054511041" Nov 03 20:45:04.071744 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.069085 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:04.489948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.489917 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" event={"ID":"b2423b82-2e79-4702-9a43-4e246f7eab55","Type":"ContainerStarted","Data":"8b351212c2fc23568c25fe180a874977044f9d98091a86e2a499eeceaa18ce57"} Nov 03 20:45:04.492051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.492028 4711 generic.go:356] "Generic (PLEG): container finished" podID="9bd49149-1509-4696-8418-bfe975d02584" containerID="46f94a32534612a58213428f187ff364a22ff608d6b2e93e0d466459bc306219" exitCode=60 Nov 03 20:45:04.492162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.492108 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" event={"ID":"9bd49149-1509-4696-8418-bfe975d02584","Type":"ContainerDied","Data":"46f94a32534612a58213428f187ff364a22ff608d6b2e93e0d466459bc306219"} Nov 03 20:45:04.492227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.492166 4711 scope.go:122] "RemoveContainer" containerID="76d07e7ef38d6b59ca2fb02caaeabba2c1d531bfdbf008bf7705ef8c38fe5626" Nov 03 20:45:04.492396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.492378 4711 scope.go:122] "RemoveContainer" containerID="46f94a32534612a58213428f187ff364a22ff608d6b2e93e0d466459bc306219" Nov 03 20:45:04.492744 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:04.492715 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf_odh-ai-gateway-infra(9bd49149-1509-4696-8418-bfe975d02584)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" podUID="9bd49149-1509-4696-8418-bfe975d02584" Nov 03 20:45:04.494389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.494369 4711 generic.go:356] "Generic (PLEG): container finished" podID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerID="95941d65586fc38d90b89010a7c04047d2da2cfa068aeb59c09ccefbe8d07803" exitCode=60 Nov 03 20:45:04.494510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.494461 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" event={"ID":"4b0b88d8-6cbf-4851-a3ec-412c44929ffc","Type":"ContainerDied","Data":"95941d65586fc38d90b89010a7c04047d2da2cfa068aeb59c09ccefbe8d07803"} Nov 03 20:45:04.494712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.494698 4711 scope.go:122] "RemoveContainer" containerID="95941d65586fc38d90b89010a7c04047d2da2cfa068aeb59c09ccefbe8d07803" Nov 03 20:45:04.494992 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:04.494967 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl_odh-ai-gateway-infra(4b0b88d8-6cbf-4851-a3ec-412c44929ffc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" Nov 03 20:45:04.496560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.496538 4711 generic.go:356] "Generic (PLEG): container finished" podID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerID="6ab33ec170abb99e410ba1a940a1dc46b2cb2e2794acca9ee1d90ea499b7730e" exitCode=60 Nov 03 20:45:04.496642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.496619 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" event={"ID":"e41bb147-1f27-47bb-a3be-4c3538170edd","Type":"ContainerDied","Data":"6ab33ec170abb99e410ba1a940a1dc46b2cb2e2794acca9ee1d90ea499b7730e"} Nov 03 20:45:04.496862 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.496838 4711 scope.go:122] "RemoveContainer" containerID="6ab33ec170abb99e410ba1a940a1dc46b2cb2e2794acca9ee1d90ea499b7730e" Nov 03 20:45:04.497109 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:04.497091 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq_odh-ai-gateway-infra(e41bb147-1f27-47bb-a3be-4c3538170edd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" Nov 03 20:45:04.498796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.498776 4711 generic.go:356] "Generic (PLEG): container finished" podID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerID="339e40408cad03bc718a46bd53aeeb0bf0747ec17fae264952fd6f17068dc077" exitCode=60 Nov 03 20:45:04.498988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.498967 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" event={"ID":"8f40772e-a2fa-4b57-ab5a-64a4d571ab88","Type":"ContainerDied","Data":"339e40408cad03bc718a46bd53aeeb0bf0747ec17fae264952fd6f17068dc077"} Nov 03 20:45:04.499152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.499137 4711 scope.go:122] "RemoveContainer" containerID="339e40408cad03bc718a46bd53aeeb0bf0747ec17fae264952fd6f17068dc077" Nov 03 20:45:04.499414 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:04.499390 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg_odh-ai-gateway-infra(8f40772e-a2fa-4b57-ab5a-64a4d571ab88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" Nov 03 20:45:04.501251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.501210 4711 generic.go:356] "Generic (PLEG): container finished" podID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerID="204c7c316137073aefa7adacd43450e13eb67e90c665531679592563f2ea2924" exitCode=60 Nov 03 20:45:04.501355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.501295 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" event={"ID":"fc6aeae1-eeef-47ee-8798-438f7577a6c1","Type":"ContainerDied","Data":"204c7c316137073aefa7adacd43450e13eb67e90c665531679592563f2ea2924"} Nov 03 20:45:04.503427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.503400 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" event={"ID":"fd0c591c-e5ea-4e21-89ec-b8aee7045acb","Type":"ContainerStarted","Data":"3b48567ebd37bd12e4360ccd5b5e782bdc2326508c00ee52f09d7a87a6c54886"} Nov 03 20:45:04.503635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.503619 4711 scope.go:122] "RemoveContainer" containerID="204c7c316137073aefa7adacd43450e13eb67e90c665531679592563f2ea2924" Nov 03 20:45:04.503905 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:04.503884 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-wmz8z_odh-ai-gateway-infra(fc6aeae1-eeef-47ee-8798-438f7577a6c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" Nov 03 20:45:04.548226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.548201 4711 scope.go:122] "RemoveContainer" containerID="5daa11b635cc0c08ae17ef432d104485647edb1efebbf20bebe5e5fbc65fc0e6" Nov 03 20:45:04.562686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.562594 4711 scope.go:122] "RemoveContainer" containerID="67afcc17f4cc22a14bb4685f0a00d1c2821f3e845ea3d4e4ec0e4b9ac04db593" Nov 03 20:45:04.576539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.576515 4711 scope.go:122] "RemoveContainer" containerID="400dd7b7cd31ef5f9b492f4d7b795e8d892b773b6623c4bfec4d62a84a5b0260" Nov 03 20:45:04.590537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.590518 4711 scope.go:122] "RemoveContainer" containerID="87e1e12553ba0405a71cd0ee594f1e1d07c946b8270e9fb2e8f6723ac785b675" Nov 03 20:45:04.834599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:04.832865 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:05.518005 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.517771 4711 generic.go:356] "Generic (PLEG): container finished" podID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerID="8b351212c2fc23568c25fe180a874977044f9d98091a86e2a499eeceaa18ce57" exitCode=60 Nov 03 20:45:05.518005 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.517814 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" event={"ID":"b2423b82-2e79-4702-9a43-4e246f7eab55","Type":"ContainerDied","Data":"8b351212c2fc23568c25fe180a874977044f9d98091a86e2a499eeceaa18ce57"} Nov 03 20:45:05.518005 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.517888 4711 scope.go:122] "RemoveContainer" containerID="8849207a4546cd3ee3a08e0a5bf7d42469794b551513b4db2d78324c97547029" Nov 03 20:45:05.518702 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.518592 4711 scope.go:122] "RemoveContainer" containerID="8b351212c2fc23568c25fe180a874977044f9d98091a86e2a499eeceaa18ce57" Nov 03 20:45:05.519103 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:05.519069 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf_odh-ai-gateway-infra(b2423b82-2e79-4702-9a43-4e246f7eab55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" Nov 03 20:45:05.520874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.520852 4711 scope.go:122] "RemoveContainer" containerID="46f94a32534612a58213428f187ff364a22ff608d6b2e93e0d466459bc306219" Nov 03 20:45:05.521435 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:05.521288 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf_odh-ai-gateway-infra(9bd49149-1509-4696-8418-bfe975d02584)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" podUID="9bd49149-1509-4696-8418-bfe975d02584" Nov 03 20:45:05.525625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.525608 4711 scope.go:122] "RemoveContainer" containerID="95941d65586fc38d90b89010a7c04047d2da2cfa068aeb59c09ccefbe8d07803" Nov 03 20:45:05.527208 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:05.527179 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl_odh-ai-gateway-infra(4b0b88d8-6cbf-4851-a3ec-412c44929ffc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" Nov 03 20:45:05.529116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.529093 4711 scope.go:122] "RemoveContainer" containerID="6ab33ec170abb99e410ba1a940a1dc46b2cb2e2794acca9ee1d90ea499b7730e" Nov 03 20:45:05.529493 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:05.529472 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq_odh-ai-gateway-infra(e41bb147-1f27-47bb-a3be-4c3538170edd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" Nov 03 20:45:05.532338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.532315 4711 scope.go:122] "RemoveContainer" containerID="339e40408cad03bc718a46bd53aeeb0bf0747ec17fae264952fd6f17068dc077" Nov 03 20:45:05.532823 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:05.532796 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg_odh-ai-gateway-infra(8f40772e-a2fa-4b57-ab5a-64a4d571ab88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" Nov 03 20:45:05.535227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.534819 4711 scope.go:122] "RemoveContainer" containerID="204c7c316137073aefa7adacd43450e13eb67e90c665531679592563f2ea2924" Nov 03 20:45:05.535227 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:05.535117 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-wmz8z_odh-ai-gateway-infra(fc6aeae1-eeef-47ee-8798-438f7577a6c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" Nov 03 20:45:05.537994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.537970 4711 generic.go:356] "Generic (PLEG): container finished" podID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerID="3b48567ebd37bd12e4360ccd5b5e782bdc2326508c00ee52f09d7a87a6c54886" exitCode=60 Nov 03 20:45:05.538087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.538020 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" event={"ID":"fd0c591c-e5ea-4e21-89ec-b8aee7045acb","Type":"ContainerDied","Data":"3b48567ebd37bd12e4360ccd5b5e782bdc2326508c00ee52f09d7a87a6c54886"} Nov 03 20:45:05.538328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.538307 4711 scope.go:122] "RemoveContainer" containerID="3b48567ebd37bd12e4360ccd5b5e782bdc2326508c00ee52f09d7a87a6c54886" Nov 03 20:45:05.538793 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:05.538693 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk_odh-ai-gateway-infra(fd0c591c-e5ea-4e21-89ec-b8aee7045acb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" Nov 03 20:45:05.545272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.545251 4711 scope.go:122] "RemoveContainer" containerID="41eb1d04c3e593d9f4d668a61535ed3174804b780bff0863ed0240967aba583d" Nov 03 20:45:05.592995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.592961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:05.738345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.737938 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:05.886299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:05.886208 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:06.551567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:06.551535 4711 scope.go:122] "RemoveContainer" containerID="3b48567ebd37bd12e4360ccd5b5e782bdc2326508c00ee52f09d7a87a6c54886" Nov 03 20:45:06.552072 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:06.552027 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk_odh-ai-gateway-infra(fd0c591c-e5ea-4e21-89ec-b8aee7045acb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" Nov 03 20:45:06.554590 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:06.554569 4711 scope.go:122] "RemoveContainer" containerID="8b351212c2fc23568c25fe180a874977044f9d98091a86e2a499eeceaa18ce57" Nov 03 20:45:06.555054 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:06.555020 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf_odh-ai-gateway-infra(b2423b82-2e79-4702-9a43-4e246f7eab55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" Nov 03 20:45:06.707607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:06.707557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:07.731740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:07.731704 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:08.412638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:08.412265 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:09.119720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:09.119679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:09.886469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:09.885743 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:10.555500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:10.555463 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:11.299229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:11.299181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:12.009934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:12.009894 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:12.696550 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:12.696518 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:13.423303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:13.423265 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:14.106846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:14.106808 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:14.677059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:14.677024 4711 scope.go:122] "RemoveContainer" containerID="6ab33ec170abb99e410ba1a940a1dc46b2cb2e2794acca9ee1d90ea499b7730e" Nov 03 20:45:15.002198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:15.000019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:15.673391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:15.673302 4711 scope.go:122] "RemoveContainer" containerID="95941d65586fc38d90b89010a7c04047d2da2cfa068aeb59c09ccefbe8d07803" Nov 03 20:45:15.673712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:15.673692 4711 scope.go:122] "RemoveContainer" containerID="204c7c316137073aefa7adacd43450e13eb67e90c665531679592563f2ea2924" Nov 03 20:45:15.735115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:15.734787 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:15.736937 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:15.736907 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" event={"ID":"e41bb147-1f27-47bb-a3be-4c3538170edd","Type":"ContainerStarted","Data":"11b4e40014d6f59a359fab02b4f32bfc015851ec14d37eafd66452f4f8ab9151"} Nov 03 20:45:16.496380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.496291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:16.681584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.681553 4711 scope.go:122] "RemoveContainer" containerID="46f94a32534612a58213428f187ff364a22ff608d6b2e93e0d466459bc306219" Nov 03 20:45:16.681770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.681752 4711 scope.go:122] "RemoveContainer" containerID="8b351212c2fc23568c25fe180a874977044f9d98091a86e2a499eeceaa18ce57" Nov 03 20:45:16.681906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.681879 4711 scope.go:122] "RemoveContainer" containerID="339e40408cad03bc718a46bd53aeeb0bf0747ec17fae264952fd6f17068dc077" Nov 03 20:45:16.747015 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.746937 4711 generic.go:356] "Generic (PLEG): container finished" podID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerID="0cd38f73589687fe65f96a6c3649d5e87dcd3df66303fe840b1efc3835cd779d" exitCode=60 Nov 03 20:45:16.747519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.747013 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" event={"ID":"4b0b88d8-6cbf-4851-a3ec-412c44929ffc","Type":"ContainerDied","Data":"0cd38f73589687fe65f96a6c3649d5e87dcd3df66303fe840b1efc3835cd779d"} Nov 03 20:45:16.747519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.747067 4711 scope.go:122] "RemoveContainer" containerID="95941d65586fc38d90b89010a7c04047d2da2cfa068aeb59c09ccefbe8d07803" Nov 03 20:45:16.747672 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.747588 4711 scope.go:122] "RemoveContainer" containerID="0cd38f73589687fe65f96a6c3649d5e87dcd3df66303fe840b1efc3835cd779d" Nov 03 20:45:16.748064 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:16.748033 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl_odh-ai-gateway-infra(4b0b88d8-6cbf-4851-a3ec-412c44929ffc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" Nov 03 20:45:16.750047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.750026 4711 generic.go:356] "Generic (PLEG): container finished" podID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerID="11b4e40014d6f59a359fab02b4f32bfc015851ec14d37eafd66452f4f8ab9151" exitCode=60 Nov 03 20:45:16.750151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.750091 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" event={"ID":"e41bb147-1f27-47bb-a3be-4c3538170edd","Type":"ContainerDied","Data":"11b4e40014d6f59a359fab02b4f32bfc015851ec14d37eafd66452f4f8ab9151"} Nov 03 20:45:16.750596 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.750577 4711 scope.go:122] "RemoveContainer" containerID="11b4e40014d6f59a359fab02b4f32bfc015851ec14d37eafd66452f4f8ab9151" Nov 03 20:45:16.751005 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:16.750979 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq_odh-ai-gateway-infra(e41bb147-1f27-47bb-a3be-4c3538170edd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" Nov 03 20:45:16.752114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.752095 4711 generic.go:356] "Generic (PLEG): container finished" podID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerID="0b6e2c32e4b6993671b5a455db1e6d292f2df81b14fe1f12a26b1d4e10653048" exitCode=60 Nov 03 20:45:16.752229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.752146 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" event={"ID":"fc6aeae1-eeef-47ee-8798-438f7577a6c1","Type":"ContainerDied","Data":"0b6e2c32e4b6993671b5a455db1e6d292f2df81b14fe1f12a26b1d4e10653048"} Nov 03 20:45:16.752553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.752536 4711 scope.go:122] "RemoveContainer" containerID="0b6e2c32e4b6993671b5a455db1e6d292f2df81b14fe1f12a26b1d4e10653048" Nov 03 20:45:16.752862 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:16.752842 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-wmz8z_odh-ai-gateway-infra(fc6aeae1-eeef-47ee-8798-438f7577a6c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" Nov 03 20:45:16.784729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.784698 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq"] Nov 03 20:45:16.834065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.834041 4711 scope.go:122] "RemoveContainer" containerID="6ab33ec170abb99e410ba1a940a1dc46b2cb2e2794acca9ee1d90ea499b7730e" Nov 03 20:45:16.850350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:16.850329 4711 scope.go:122] "RemoveContainer" containerID="204c7c316137073aefa7adacd43450e13eb67e90c665531679592563f2ea2924" Nov 03 20:45:17.295063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.295031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:17.436564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.432171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:17.535332 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:17.535294 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:45:17.673276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.672604 4711 scope.go:122] "RemoveContainer" containerID="3b48567ebd37bd12e4360ccd5b5e782bdc2326508c00ee52f09d7a87a6c54886" Nov 03 20:45:17.763118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.763078 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" event={"ID":"8f40772e-a2fa-4b57-ab5a-64a4d571ab88","Type":"ContainerStarted","Data":"3f2fe97403ea4f750e4ef690cda1f1e1fac09a2aa0204e5543e06f5547103a23"} Nov 03 20:45:17.768135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.768110 4711 generic.go:356] "Generic (PLEG): container finished" podID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerID="287383a9c7be417b1b3fe8dd827c3a6872c0f6d8855edb62942763ca482a37a8" exitCode=60 Nov 03 20:45:17.768260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.768181 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" event={"ID":"b2423b82-2e79-4702-9a43-4e246f7eab55","Type":"ContainerDied","Data":"287383a9c7be417b1b3fe8dd827c3a6872c0f6d8855edb62942763ca482a37a8"} Nov 03 20:45:17.768260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.768236 4711 scope.go:122] "RemoveContainer" containerID="8b351212c2fc23568c25fe180a874977044f9d98091a86e2a499eeceaa18ce57" Nov 03 20:45:17.768732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.768710 4711 scope.go:122] "RemoveContainer" containerID="287383a9c7be417b1b3fe8dd827c3a6872c0f6d8855edb62942763ca482a37a8" Nov 03 20:45:17.769177 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:17.769136 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf_odh-ai-gateway-infra(b2423b82-2e79-4702-9a43-4e246f7eab55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" Nov 03 20:45:17.771487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.771459 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" event={"ID":"9bd49149-1509-4696-8418-bfe975d02584","Type":"ContainerStarted","Data":"79f2e805dca0dcc427f6fd6ce97f4e18741fd03c3876dbf9e4f9154c817f106c"} Nov 03 20:45:17.780870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.780844 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl"] Nov 03 20:45:17.802347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:17.802313 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z"] Nov 03 20:45:18.021459 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.021407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:18.108895 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.108143 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:18.238518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.190116 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41bb147-1f27-47bb-a3be-4c3538170edd-kube-api-access-bnhw9\" (UniqueName: \"kubernetes.io/projected/e41bb147-1f27-47bb-a3be-4c3538170edd-kube-api-access-bnhw9\") pod \"e41bb147-1f27-47bb-a3be-4c3538170edd\" (UID: \"e41bb147-1f27-47bb-a3be-4c3538170edd\") " Nov 03 20:45:18.238518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.190338 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41bb147-1f27-47bb-a3be-4c3538170edd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41bb147-1f27-47bb-a3be-4c3538170edd-maas-api-service-ca\") pod \"e41bb147-1f27-47bb-a3be-4c3538170edd\" (UID: \"e41bb147-1f27-47bb-a3be-4c3538170edd\") " Nov 03 20:45:18.238518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.233271 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41bb147-1f27-47bb-a3be-4c3538170edd-maas-api-service-ca" pod "e41bb147-1f27-47bb-a3be-4c3538170edd" (UID: "e41bb147-1f27-47bb-a3be-4c3538170edd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:18.245767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.245718 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e41bb147-1f27-47bb-a3be-4c3538170edd-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:18.246193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.246114 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:18.259537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.258017 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41bb147-1f27-47bb-a3be-4c3538170edd-kube-api-access-bnhw9" pod "e41bb147-1f27-47bb-a3be-4c3538170edd" (UID: "e41bb147-1f27-47bb-a3be-4c3538170edd"). InnerVolumeSpecName "kube-api-access-bnhw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:18.286417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.286081 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:18.346803 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.346765 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-maas-api-service-ca\") pod \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\" (UID: \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\") " Nov 03 20:45:18.347022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.347004 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc6aeae1-eeef-47ee-8798-438f7577a6c1-kube-api-access-2srh7\" (UniqueName: \"kubernetes.io/projected/fc6aeae1-eeef-47ee-8798-438f7577a6c1-kube-api-access-2srh7\") pod \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\" (UID: \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\") " Nov 03 20:45:18.347202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.347173 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc6aeae1-eeef-47ee-8798-438f7577a6c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6aeae1-eeef-47ee-8798-438f7577a6c1-maas-api-service-ca\") pod \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\" (UID: \"fc6aeae1-eeef-47ee-8798-438f7577a6c1\") " Nov 03 20:45:18.347408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.347386 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-kube-api-access-ls8nc\" (UniqueName: \"kubernetes.io/projected/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-kube-api-access-ls8nc\") pod \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\" (UID: \"4b0b88d8-6cbf-4851-a3ec-412c44929ffc\") " Nov 03 20:45:18.347971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.347948 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-maas-api-service-ca" pod "4b0b88d8-6cbf-4851-a3ec-412c44929ffc" (UID: "4b0b88d8-6cbf-4851-a3ec-412c44929ffc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:18.348275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.348249 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc6aeae1-eeef-47ee-8798-438f7577a6c1-maas-api-service-ca" pod "fc6aeae1-eeef-47ee-8798-438f7577a6c1" (UID: "fc6aeae1-eeef-47ee-8798-438f7577a6c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:18.348864 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.348841 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc6aeae1-eeef-47ee-8798-438f7577a6c1-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:18.348964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.348868 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:18.348964 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.348883 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhw9\" (UniqueName: \"kubernetes.io/projected/e41bb147-1f27-47bb-a3be-4c3538170edd-kube-api-access-bnhw9\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:18.350327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.350302 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc6aeae1-eeef-47ee-8798-438f7577a6c1-kube-api-access-2srh7" pod "fc6aeae1-eeef-47ee-8798-438f7577a6c1" (UID: "fc6aeae1-eeef-47ee-8798-438f7577a6c1"). InnerVolumeSpecName "kube-api-access-2srh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:18.350425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.350353 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-kube-api-access-ls8nc" pod "4b0b88d8-6cbf-4851-a3ec-412c44929ffc" (UID: "4b0b88d8-6cbf-4851-a3ec-412c44929ffc"). InnerVolumeSpecName "kube-api-access-ls8nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:18.450379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.450338 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls8nc\" (UniqueName: \"kubernetes.io/projected/4b0b88d8-6cbf-4851-a3ec-412c44929ffc-kube-api-access-ls8nc\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:18.450379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.450370 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2srh7\" (UniqueName: \"kubernetes.io/projected/fc6aeae1-eeef-47ee-8798-438f7577a6c1-kube-api-access-2srh7\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:18.795737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.795706 4711 generic.go:356] "Generic (PLEG): container finished" podID="9bd49149-1509-4696-8418-bfe975d02584" containerID="79f2e805dca0dcc427f6fd6ce97f4e18741fd03c3876dbf9e4f9154c817f106c" exitCode=60 Nov 03 20:45:18.796232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.795783 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" event={"ID":"9bd49149-1509-4696-8418-bfe975d02584","Type":"ContainerDied","Data":"79f2e805dca0dcc427f6fd6ce97f4e18741fd03c3876dbf9e4f9154c817f106c"} Nov 03 20:45:18.796232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.795853 4711 scope.go:122] "RemoveContainer" containerID="46f94a32534612a58213428f187ff364a22ff608d6b2e93e0d466459bc306219" Nov 03 20:45:18.796348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.796284 4711 scope.go:122] "RemoveContainer" containerID="79f2e805dca0dcc427f6fd6ce97f4e18741fd03c3876dbf9e4f9154c817f106c" Nov 03 20:45:18.796813 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:18.796789 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf_odh-ai-gateway-infra(9bd49149-1509-4696-8418-bfe975d02584)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" podUID="9bd49149-1509-4696-8418-bfe975d02584" Nov 03 20:45:18.797615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.797599 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" Nov 03 20:45:18.797717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.797645 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl" event={"ID":"4b0b88d8-6cbf-4851-a3ec-412c44929ffc","Type":"ContainerDied","Data":"8cdd08a1d217c4004193aba0a2c17170382dd9dfb23d6b2efa676699d92f95e1"} Nov 03 20:45:18.798302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.798277 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg"] Nov 03 20:45:18.799134 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.799119 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" Nov 03 20:45:18.799263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.799163 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq" event={"ID":"e41bb147-1f27-47bb-a3be-4c3538170edd","Type":"ContainerDied","Data":"0b33b7fb229e4e80801fa437c8d1c635875c92bb69f39e6b4156fdc662e8b771"} Nov 03 20:45:18.801417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.801395 4711 generic.go:356] "Generic (PLEG): container finished" podID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerID="3f2fe97403ea4f750e4ef690cda1f1e1fac09a2aa0204e5543e06f5547103a23" exitCode=60 Nov 03 20:45:18.801650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.801472 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" event={"ID":"8f40772e-a2fa-4b57-ab5a-64a4d571ab88","Type":"ContainerDied","Data":"3f2fe97403ea4f750e4ef690cda1f1e1fac09a2aa0204e5543e06f5547103a23"} Nov 03 20:45:18.810218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.810195 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" Nov 03 20:45:18.810418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.810233 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z" event={"ID":"fc6aeae1-eeef-47ee-8798-438f7577a6c1","Type":"ContainerDied","Data":"3d205b2d96d7d9b902d9c6b2cbe425950901dbed4405c9f2131a13193f679f26"} Nov 03 20:45:18.813421 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.813400 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" event={"ID":"fd0c591c-e5ea-4e21-89ec-b8aee7045acb","Type":"ContainerStarted","Data":"c35fe9c57b2db1b9cfaa5928c11a20907d36fd848ef2e03f9d726b1af506d45e"} Nov 03 20:45:18.816625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.816601 4711 scope.go:122] "RemoveContainer" containerID="0cd38f73589687fe65f96a6c3649d5e87dcd3df66303fe840b1efc3835cd779d" Nov 03 20:45:18.891827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.858166 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:18.910533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.907133 4711 scope.go:122] "RemoveContainer" containerID="11b4e40014d6f59a359fab02b4f32bfc015851ec14d37eafd66452f4f8ab9151" Nov 03 20:45:18.935319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.934431 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf"] Nov 03 20:45:18.943001 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.942563 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z"] Nov 03 20:45:18.952247 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.950643 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-wmz8z"] Nov 03 20:45:18.967078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.967047 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf"] Nov 03 20:45:18.979112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.979069 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq"] Nov 03 20:45:18.985496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.985203 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8c6934-29895645-j4rsq"] Nov 03 20:45:18.994104 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:18.994082 4711 scope.go:122] "RemoveContainer" containerID="339e40408cad03bc718a46bd53aeeb0bf0747ec17fae264952fd6f17068dc077" Nov 03 20:45:19.019600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.019577 4711 scope.go:122] "RemoveContainer" containerID="0b6e2c32e4b6993671b5a455db1e6d292f2df81b14fe1f12a26b1d4e10653048" Nov 03 20:45:19.030502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.030412 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl"] Nov 03 20:45:19.040192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.039970 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-ba89e0-29895645-vvkbl"] Nov 03 20:45:19.170944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.170911 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:19.216907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.216872 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:19.271632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.271594 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2423b82-2e79-4702-9a43-4e246f7eab55-kube-api-access-p6fsn\" (UniqueName: \"kubernetes.io/projected/b2423b82-2e79-4702-9a43-4e246f7eab55-kube-api-access-p6fsn\") pod \"b2423b82-2e79-4702-9a43-4e246f7eab55\" (UID: \"b2423b82-2e79-4702-9a43-4e246f7eab55\") " Nov 03 20:45:19.271833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.271702 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-maas-api-service-ca\") pod \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\" (UID: \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\") " Nov 03 20:45:19.271833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.271780 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2423b82-2e79-4702-9a43-4e246f7eab55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2423b82-2e79-4702-9a43-4e246f7eab55-maas-api-service-ca\") pod \"b2423b82-2e79-4702-9a43-4e246f7eab55\" (UID: \"b2423b82-2e79-4702-9a43-4e246f7eab55\") " Nov 03 20:45:19.271962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.271907 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-kube-api-access-wlpws\" (UniqueName: \"kubernetes.io/projected/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-kube-api-access-wlpws\") pod \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\" (UID: \"8f40772e-a2fa-4b57-ab5a-64a4d571ab88\") " Nov 03 20:45:19.272781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.272733 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-maas-api-service-ca" pod "8f40772e-a2fa-4b57-ab5a-64a4d571ab88" (UID: "8f40772e-a2fa-4b57-ab5a-64a4d571ab88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:19.273217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.273186 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2423b82-2e79-4702-9a43-4e246f7eab55-maas-api-service-ca" pod "b2423b82-2e79-4702-9a43-4e246f7eab55" (UID: "b2423b82-2e79-4702-9a43-4e246f7eab55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:19.275096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.275069 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-kube-api-access-wlpws" pod "8f40772e-a2fa-4b57-ab5a-64a4d571ab88" (UID: "8f40772e-a2fa-4b57-ab5a-64a4d571ab88"). InnerVolumeSpecName "kube-api-access-wlpws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:19.275296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.275271 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2423b82-2e79-4702-9a43-4e246f7eab55-kube-api-access-p6fsn" pod "b2423b82-2e79-4702-9a43-4e246f7eab55" (UID: "b2423b82-2e79-4702-9a43-4e246f7eab55"). InnerVolumeSpecName "kube-api-access-p6fsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:19.375201 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.375160 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6fsn\" (UniqueName: \"kubernetes.io/projected/b2423b82-2e79-4702-9a43-4e246f7eab55-kube-api-access-p6fsn\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:19.375201 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.375203 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:19.375495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.375248 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2423b82-2e79-4702-9a43-4e246f7eab55-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:19.375495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.375262 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlpws\" (UniqueName: \"kubernetes.io/projected/8f40772e-a2fa-4b57-ab5a-64a4d571ab88-kube-api-access-wlpws\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:19.479088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.478990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:19.792933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.792896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:19.836385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.834568 4711 generic.go:356] "Generic (PLEG): container finished" podID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerID="c35fe9c57b2db1b9cfaa5928c11a20907d36fd848ef2e03f9d726b1af506d45e" exitCode=60 Nov 03 20:45:19.836385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.834699 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" event={"ID":"fd0c591c-e5ea-4e21-89ec-b8aee7045acb","Type":"ContainerDied","Data":"c35fe9c57b2db1b9cfaa5928c11a20907d36fd848ef2e03f9d726b1af506d45e"} Nov 03 20:45:19.836385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.834761 4711 scope.go:122] "RemoveContainer" containerID="3b48567ebd37bd12e4360ccd5b5e782bdc2326508c00ee52f09d7a87a6c54886" Nov 03 20:45:19.836385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.835306 4711 scope.go:122] "RemoveContainer" containerID="c35fe9c57b2db1b9cfaa5928c11a20907d36fd848ef2e03f9d726b1af506d45e" Nov 03 20:45:19.836385 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:19.835800 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk_odh-ai-gateway-infra(fd0c591c-e5ea-4e21-89ec-b8aee7045acb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" Nov 03 20:45:19.851480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.851273 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" event={"ID":"b2423b82-2e79-4702-9a43-4e246f7eab55","Type":"ContainerDied","Data":"7505c7820ccef6253d832d6fc4422a3408df2ff6a3c228452c86a8f29e288461"} Nov 03 20:45:19.851480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.851404 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf" Nov 03 20:45:19.906268 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.905988 4711 scope.go:122] "RemoveContainer" containerID="287383a9c7be417b1b3fe8dd827c3a6872c0f6d8855edb62942763ca482a37a8" Nov 03 20:45:19.909059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.909028 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" event={"ID":"8f40772e-a2fa-4b57-ab5a-64a4d571ab88","Type":"ContainerDied","Data":"3f2b3025c7c70a2cf0c582707bd99a5a81cf7f8ffe4ca79665f62b6f6c1d2548"} Nov 03 20:45:19.909370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.909315 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg" Nov 03 20:45:19.960617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.960575 4711 scope.go:122] "RemoveContainer" containerID="3f2fe97403ea4f750e4ef690cda1f1e1fac09a2aa0204e5543e06f5547103a23" Nov 03 20:45:19.969956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.969926 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf"] Nov 03 20:45:19.978322 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.978235 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk"] Nov 03 20:45:19.995507 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.993729 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-466f83-29895645-254tf"] Nov 03 20:45:19.999034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:19.998997 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg"] Nov 03 20:45:20.015168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.015140 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-99a8fe-29895645-ps9zg"] Nov 03 20:45:20.137115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.137087 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:20.313060 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.313019 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd49149-1509-4696-8418-bfe975d02584-kube-api-access-68rj6\" (UniqueName: \"kubernetes.io/projected/9bd49149-1509-4696-8418-bfe975d02584-kube-api-access-68rj6\") pod \"9bd49149-1509-4696-8418-bfe975d02584\" (UID: \"9bd49149-1509-4696-8418-bfe975d02584\") " Nov 03 20:45:20.313270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.313139 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd49149-1509-4696-8418-bfe975d02584-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd49149-1509-4696-8418-bfe975d02584-maas-api-service-ca\") pod \"9bd49149-1509-4696-8418-bfe975d02584\" (UID: \"9bd49149-1509-4696-8418-bfe975d02584\") " Nov 03 20:45:20.313584 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.313560 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd49149-1509-4696-8418-bfe975d02584-maas-api-service-ca" pod "9bd49149-1509-4696-8418-bfe975d02584" (UID: "9bd49149-1509-4696-8418-bfe975d02584"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:20.313957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.313942 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd49149-1509-4696-8418-bfe975d02584-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:20.315212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.315186 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd49149-1509-4696-8418-bfe975d02584-kube-api-access-68rj6" pod "9bd49149-1509-4696-8418-bfe975d02584" (UID: "9bd49149-1509-4696-8418-bfe975d02584"). InnerVolumeSpecName "kube-api-access-68rj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:20.416339 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.416243 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rj6\" (UniqueName: \"kubernetes.io/projected/9bd49149-1509-4696-8418-bfe975d02584-kube-api-access-68rj6\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:20.607986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.607948 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:20.692374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.692288 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" path="/var/lib/kubelet/pods/4b0b88d8-6cbf-4851-a3ec-412c44929ffc/volumes" Nov 03 20:45:20.692976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.692953 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" path="/var/lib/kubelet/pods/8f40772e-a2fa-4b57-ab5a-64a4d571ab88/volumes" Nov 03 20:45:20.693575 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.693546 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" path="/var/lib/kubelet/pods/b2423b82-2e79-4702-9a43-4e246f7eab55/volumes" Nov 03 20:45:20.694347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.694324 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" path="/var/lib/kubelet/pods/e41bb147-1f27-47bb-a3be-4c3538170edd/volumes" Nov 03 20:45:20.694826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:20.694800 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" path="/var/lib/kubelet/pods/fc6aeae1-eeef-47ee-8798-438f7577a6c1/volumes" Nov 03 20:45:21.037467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.029200 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" event={"ID":"9bd49149-1509-4696-8418-bfe975d02584","Type":"ContainerDied","Data":"9e939587fefe2e272cbf9e23e4431fa3cc461644e011ce2523a0130a8232f4fe"} Nov 03 20:45:21.037467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.029275 4711 scope.go:122] "RemoveContainer" containerID="79f2e805dca0dcc427f6fd6ce97f4e18741fd03c3876dbf9e4f9154c817f106c" Nov 03 20:45:21.037467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.029383 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf" Nov 03 20:45:21.134632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.134606 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf"] Nov 03 20:45:21.140617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.140587 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895645-q54zf"] Nov 03 20:45:21.298963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.298934 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:21.456543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.456510 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-kube-api-access-mxskp\" (UniqueName: \"kubernetes.io/projected/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-kube-api-access-mxskp\") pod \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\" (UID: \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\") " Nov 03 20:45:21.456834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.456807 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-maas-api-service-ca\") pod \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\" (UID: \"fd0c591c-e5ea-4e21-89ec-b8aee7045acb\") " Nov 03 20:45:21.457280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.457247 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-maas-api-service-ca" pod "fd0c591c-e5ea-4e21-89ec-b8aee7045acb" (UID: "fd0c591c-e5ea-4e21-89ec-b8aee7045acb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:21.457562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.457539 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.459223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.459195 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-kube-api-access-mxskp" pod "fd0c591c-e5ea-4e21-89ec-b8aee7045acb" (UID: "fd0c591c-e5ea-4e21-89ec-b8aee7045acb"). InnerVolumeSpecName "kube-api-access-mxskp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:21.559608 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.559525 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxskp\" (UniqueName: \"kubernetes.io/projected/fd0c591c-e5ea-4e21-89ec-b8aee7045acb-kube-api-access-mxskp\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:21.620606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:21.620569 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:22.299130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:22.299107 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" Nov 03 20:45:22.300168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:22.299515 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk" event={"ID":"fd0c591c-e5ea-4e21-89ec-b8aee7045acb","Type":"ContainerDied","Data":"a07f4d345795a469523d84c0355cfbbdf9d976b99a7cf9e00a27116cb4b93843"} Nov 03 20:45:22.300270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:22.300213 4711 scope.go:122] "RemoveContainer" containerID="c35fe9c57b2db1b9cfaa5928c11a20907d36fd848ef2e03f9d726b1af506d45e" Nov 03 20:45:22.330364 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:22.330330 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk"] Nov 03 20:45:22.336562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:22.336535 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3631f-29895645-lwfwk"] Nov 03 20:45:22.657566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:22.653028 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:22.692136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:22.692102 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd49149-1509-4696-8418-bfe975d02584" path="/var/lib/kubelet/pods/9bd49149-1509-4696-8418-bfe975d02584/volumes" Nov 03 20:45:22.692748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:22.692714 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" path="/var/lib/kubelet/pods/fd0c591c-e5ea-4e21-89ec-b8aee7045acb/volumes" Nov 03 20:45:23.605470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:23.604500 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:24.529700 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:24.529669 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:25.447467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:25.435151 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:26.149305 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:26.149272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:27.101415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:27.101112 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:27.273072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:27.272351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:27.443533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:27.439877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:28.806995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:28.806952 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:29.561942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:29.561907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:30.428471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:30.427856 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:31.196178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:31.196138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:31.440255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:31.440214 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:32.276907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:32.276869 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:33.132924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:33.132885 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:33.883713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:33.883131 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:34.387142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:34.387106 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:34.816223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:34.816185 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:35.510521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:35.510480 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:36.214248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:36.214208 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:36.910512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:36.910472 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:37.056395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:37.056350 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:37.649817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:37.649777 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:37.781502 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:37.781464 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:38.119054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:38.119019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:38.990896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:38.990858 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:39.798540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:39.798504 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:40.602350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:40.602315 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:41.368835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:41.368793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:42.202140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:42.202104 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:43.122349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:43.122314 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:43.999374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:43.999339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:44.658732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:44.658695 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:45.324808 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:45.324771 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:45.694291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:45.694200 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:46.294010 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:46.293973 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:47.001538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:47.001496 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:47.535216 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:45:47.535174 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:45:47.779200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:47.779168 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:48.638217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:48.638171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:49.528910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:49.528872 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:49.617414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:49.617382 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw"] Nov 03 20:45:49.667122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:49.667084 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:49.881529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:49.852974 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:50.162920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:50.162830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:50.676158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:50.676124 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:51.585514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:51.585472 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:52.395150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:52.395117 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:53.124025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:53.123990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:53.906341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:53.906302 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:54.620123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:54.620082 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" podUID="ce91d954-23cd-40c5-a970-0fc15faafe15" containerName="payload-processing" containerID="cri-o://8cc079c5e53ff8a2128879124284ad9a798cb332548539d69aa9d940a8ed4d79" gracePeriod=25 Nov 03 20:45:54.830023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:54.829827 4711 generic.go:356] "Generic (PLEG): container finished" podID="ce91d954-23cd-40c5-a970-0fc15faafe15" containerID="8cc079c5e53ff8a2128879124284ad9a798cb332548539d69aa9d940a8ed4d79" exitCode=0 Nov 03 20:45:54.830023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:54.829971 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" event={"ID":"ce91d954-23cd-40c5-a970-0fc15faafe15","Type":"ContainerDied","Data":"8cc079c5e53ff8a2128879124284ad9a798cb332548539d69aa9d940a8ed4d79"} Nov 03 20:45:54.911383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:54.911303 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:55.056482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.055907 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:45:55.191615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.191526 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-kube-api-access-npt4l\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-kube-api-access-npt4l\") pod \"ce91d954-23cd-40c5-a970-0fc15faafe15\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " Nov 03 20:45:55.191881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.191855 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-provider-credentials\") pod \"ce91d954-23cd-40c5-a970-0fc15faafe15\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " Nov 03 20:45:55.192078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.192058 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-routing-overlay-volume\") pod \"ce91d954-23cd-40c5-a970-0fc15faafe15\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " Nov 03 20:45:55.192163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.192152 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-plugins-config-volume\") pod \"ce91d954-23cd-40c5-a970-0fc15faafe15\" (UID: \"ce91d954-23cd-40c5-a970-0fc15faafe15\") " Nov 03 20:45:55.192533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.192506 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-routing-overlay-volume" pod "ce91d954-23cd-40c5-a970-0fc15faafe15" (UID: "ce91d954-23cd-40c5-a970-0fc15faafe15"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:55.192896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.192874 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-plugins-config-volume" pod "ce91d954-23cd-40c5-a970-0fc15faafe15" (UID: "ce91d954-23cd-40c5-a970-0fc15faafe15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:55.193251 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.193227 4711 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-routing-overlay-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:55.193303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.193258 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce91d954-23cd-40c5-a970-0fc15faafe15-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:55.195747 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.195725 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-provider-credentials" pod "ce91d954-23cd-40c5-a970-0fc15faafe15" (UID: "ce91d954-23cd-40c5-a970-0fc15faafe15"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:55.198095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.198064 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-kube-api-access-npt4l" pod "ce91d954-23cd-40c5-a970-0fc15faafe15" (UID: "ce91d954-23cd-40c5-a970-0fc15faafe15"). InnerVolumeSpecName "kube-api-access-npt4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:55.294501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.294469 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npt4l\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-kube-api-access-npt4l\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:55.294501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.294498 4711 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce91d954-23cd-40c5-a970-0fc15faafe15-provider-credentials\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:55.600826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.600791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:55.838593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.838552 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" event={"ID":"ce91d954-23cd-40c5-a970-0fc15faafe15","Type":"ContainerDied","Data":"b82aced05a75c1a8eef6b7887a1e44851dcfb8f7020f57ed658afb726a3a7da5"} Nov 03 20:45:55.838593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.838589 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw" Nov 03 20:45:55.838810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.838627 4711 scope.go:122] "RemoveContainer" containerID="8cc079c5e53ff8a2128879124284ad9a798cb332548539d69aa9d940a8ed4d79" Nov 03 20:45:55.864514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.864461 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw"] Nov 03 20:45:55.895319 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:55.895284 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-6b986xfzw"] Nov 03 20:45:56.756467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:56.752706 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce91d954-23cd-40c5-a970-0fc15faafe15" path="/var/lib/kubelet/pods/ce91d954-23cd-40c5-a970-0fc15faafe15/volumes" Nov 03 20:45:57.106528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:57.106262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:57.244889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:57.244858 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:57.442966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:57.442869 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:57.826972 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:57.826940 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:57.952558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:57.952523 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:58.563896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:58.563842 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:58.697651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:58.697617 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:58.840528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:58.836627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:59.386917 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.386876 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:59.486226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.473765 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d"] Nov 03 20:45:59.495188 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495150 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ce91d954-23cd-40c5-a970-0fc15faafe15" containerName="payload-processing" Nov 03 20:45:59.495188 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495186 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce91d954-23cd-40c5-a970-0fc15faafe15" containerName="payload-processing" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495201 4711 cpu_manager.go:414] "containerMap: removing container" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495209 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495227 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495234 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495250 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495258 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495276 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495284 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495295 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495302 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495316 4711 cpu_manager.go:414] "containerMap: removing container" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495324 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495336 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495344 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495360 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495367 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495381 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495388 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495402 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9bd49149-1509-4696-8418-bfe975d02584" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495409 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd49149-1509-4696-8418-bfe975d02584" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495437 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9bd49149-1509-4696-8418-bfe975d02584" containerName="cleanup" Nov 03 20:45:59.495460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495459 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd49149-1509-4696-8418-bfe975d02584" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495475 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495483 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495506 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495513 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495537 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9bd49149-1509-4696-8418-bfe975d02584" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495544 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd49149-1509-4696-8418-bfe975d02584" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495565 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495574 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495591 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495598 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495814 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495826 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495836 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495844 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495855 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce91d954-23cd-40c5-a970-0fc15faafe15" containerName="payload-processing" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495866 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495877 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495888 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495899 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495906 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495916 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495927 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495935 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495944 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd49149-1509-4696-8418-bfe975d02584" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495955 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495963 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd49149-1509-4696-8418-bfe975d02584" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495973 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495983 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd49149-1509-4696-8418-bfe975d02584" containerName="cleanup" Nov 03 20:45:59.496593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.495994 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerName="cleanup" Nov 03 20:45:59.541484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.537408 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.563733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.558697 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d"] Nov 03 20:45:59.578483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.575766 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"openshift-service-ca.crt\"" Nov 03 20:45:59.580633 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.580608 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"payload-processing-external-model-e2e--4cfe9bda-dockercfg-5rtzb\"" Nov 03 20:45:59.581310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.581285 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"payload-processing-external-model-plugins-e2e-extmod-a-dce009\"" Nov 03 20:45:59.582221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.582204 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"kube-root-ca.crt\"" Nov 03 20:45:59.582889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.582874 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"agc-cred-fdd789046104\"" Nov 03 20:45:59.583257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.583127 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-dce009\"/\"routing-overlay\"" Nov 03 20:45:59.603047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.601716 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:45:59.636388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.636352 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbckk\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-kube-api-access-nbckk\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.636617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.636417 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.636617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.636593 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.636777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.636632 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.738096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.738005 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.738096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.738062 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.738351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.738332 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbckk\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-kube-api-access-nbckk\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.738413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.738393 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.738978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.738954 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.739213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.739192 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.741215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.741196 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.759898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.759870 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbckk\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-kube-api-access-nbckk\") pod \"payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.916238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.916207 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:45:59.922143 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:45:59.922117 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:00.321135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:00.321101 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d"] Nov 03 20:46:00.321494 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:46:00.321459 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1bfdda_f732_4b1e_bbb0_354bbfcf22fb.slice/crio-833ebd85f006c2bea21ff481d68f440b09136cf46b73c8bca03790673cd77bd1 WatchSource:0}: Error finding container 833ebd85f006c2bea21ff481d68f440b09136cf46b73c8bca03790673cd77bd1: Status 404 returned error can't find the container with id 833ebd85f006c2bea21ff481d68f440b09136cf46b73c8bca03790673cd77bd1 Nov 03 20:46:00.519323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:00.519291 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" event={"ID":"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb","Type":"ContainerStarted","Data":"ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4"} Nov 03 20:46:00.519519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:00.519345 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" event={"ID":"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb","Type":"ContainerStarted","Data":"833ebd85f006c2bea21ff481d68f440b09136cf46b73c8bca03790673cd77bd1"} Nov 03 20:46:00.519519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:00.519501 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:46:00.540320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:00.540271 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" podStartSLOduration=2.540257339 podStartE2EDuration="2.540257339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:00.53871836 +0000 UTC m=+2663.988824407" watchObservedRunningTime="2026-11-03 20:46:00.540257339 +0000 UTC m=+2663.990363384" Nov 03 20:46:00.629731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:00.629649 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:01.281638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:01.281598 4711 patch_prober.go:29] interesting pod/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:46:01.282061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:01.281664 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:46:01.457498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:01.448584 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:02.199812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:02.199773 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:02.956474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:02.952945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:03.689683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:03.689648 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:04.363082 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:04.363042 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:04.558297 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:04.558264 4711 generic.go:356] "Generic (PLEG): container finished" podID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerID="560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115" exitCode=0 Nov 03 20:46:04.558488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:04.558339 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" event={"ID":"84ed0c60-5fec-4a41-912b-f5f2e9ff1516","Type":"ContainerDied","Data":"560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115"} Nov 03 20:46:04.558841 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:04.558824 4711 scope.go:122] "RemoveContainer" containerID="560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115" Nov 03 20:46:04.560806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:04.560781 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:04.560876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:04.560849 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:04.560912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:04.560882 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:05.128851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:05.128804 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:05.585819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:05.585770 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" event={"ID":"84ed0c60-5fec-4a41-912b-f5f2e9ff1516","Type":"ContainerStarted","Data":"fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2"} Nov 03 20:46:05.627902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:05.627872 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:46:05.628080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:05.628061 4711 patch_prober.go:29] interesting pod/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:46:05.628136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:05.628116 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 20:46:05.868534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:05.865572 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:06.522852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:06.522816 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:06.630903 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:06.630870 4711 patch_prober.go:29] interesting pod/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:46:06.632137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:06.632101 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 20:46:07.262470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.259947 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:07.302262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.302225 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr"] Nov 03 20:46:07.303904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.303871 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerName="cleanup" Nov 03 20:46:07.304123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.304104 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2423b82-2e79-4702-9a43-4e246f7eab55" containerName="cleanup" Nov 03 20:46:07.304276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.304261 4711 cpu_manager.go:414] "containerMap: removing container" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerName="cleanup" Nov 03 20:46:07.304389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.304374 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerName="cleanup" Nov 03 20:46:07.312039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.304483 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerName="cleanup" Nov 03 20:46:07.312229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.312208 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41bb147-1f27-47bb-a3be-4c3538170edd" containerName="cleanup" Nov 03 20:46:07.312345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.312334 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerName="cleanup" Nov 03 20:46:07.312429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.312418 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerName="cleanup" Nov 03 20:46:07.312598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.312583 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerName="cleanup" Nov 03 20:46:07.312698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.312687 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc6aeae1-eeef-47ee-8798-438f7577a6c1" containerName="cleanup" Nov 03 20:46:07.313034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.313018 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd0c591c-e5ea-4e21-89ec-b8aee7045acb" containerName="cleanup" Nov 03 20:46:07.313145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.313134 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f40772e-a2fa-4b57-ab5a-64a4d571ab88" containerName="cleanup" Nov 03 20:46:07.313239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.313228 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b0b88d8-6cbf-4851-a3ec-412c44929ffc" containerName="cleanup" Nov 03 20:46:07.317137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.317115 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.326204 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.326162 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3d4fe2-openshift-default-dockercfg-rxt6m\"" Nov 03 20:46:07.333613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.333581 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr"] Nov 03 20:46:07.389471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.380414 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-certs\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.397476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.389787 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.397743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.397725 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-data\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.397859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.397849 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac7766-28e7-47ae-9e68-29251a841002-istiod-ca-cert\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.398135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.398125 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-envoy\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.398223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.398212 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-credential-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.398309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.398301 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-istio-token\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.398393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.398381 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac7766-28e7-47ae-9e68-29251a841002-istio-podinfo\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.398481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.398470 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzlf\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-kube-api-access-hhzlf\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500107 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-envoy\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500150 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-credential-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500184 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-istio-token\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500214 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac7766-28e7-47ae-9e68-29251a841002-istio-podinfo\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500232 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzlf\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-kube-api-access-hhzlf\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500265 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-certs\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500322 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500361 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-data\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500429 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac7766-28e7-47ae-9e68-29251a841002-istiod-ca-cert\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500718 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-credential-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.500836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500750 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-certs\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.501008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500857 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-socket\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.501008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.500982 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-data\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.501291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.501268 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac7766-28e7-47ae-9e68-29251a841002-istiod-ca-cert\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.502709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.502683 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-envoy\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.502858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.502842 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac7766-28e7-47ae-9e68-29251a841002-istio-podinfo\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.512657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.512598 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-istio-token\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.512912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.512893 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzlf\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-kube-api-access-hhzlf\") pod \"e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.628803 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.628764 4711 patch_prober.go:29] interesting pod/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:46:07.629013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.628846 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 20:46:07.659905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.659877 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:07.998607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:07.998580 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr"] Nov 03 20:46:08.001272 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:46:08.001243 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ac7766_28e7_47ae_9e68_29251a841002.slice/crio-c48f30ee9259f421b926f5c44716a40e61e0a519a7d08f72ccf2c3259b79beac WatchSource:0}: Error finding container c48f30ee9259f421b926f5c44716a40e61e0a519a7d08f72ccf2c3259b79beac: Status 404 returned error can't find the container with id c48f30ee9259f421b926f5c44716a40e61e0a519a7d08f72ccf2c3259b79beac Nov 03 20:46:08.004318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.004283 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:08.004423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.004351 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:08.004423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.004382 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:46:08.007326 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.007300 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:08.628801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.628764 4711 patch_prober.go:29] interesting pod/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:46:08.629001 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.628820 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 20:46:08.656650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.656618 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" event={"ID":"44ac7766-28e7-47ae-9e68-29251a841002","Type":"ContainerStarted","Data":"6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c"} Nov 03 20:46:08.656819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.656670 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" event={"ID":"44ac7766-28e7-47ae-9e68-29251a841002","Type":"ContainerStarted","Data":"c48f30ee9259f421b926f5c44716a40e61e0a519a7d08f72ccf2c3259b79beac"} Nov 03 20:46:08.659948 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.659929 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:08.702477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.697145 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" podStartSLOduration=1.697125795 podStartE2EDuration="1.697125795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:08.684957254 +0000 UTC m=+2672.135063300" watchObservedRunningTime="2026-11-03 20:46:08.697125795 +0000 UTC m=+2672.147231842" Nov 03 20:46:08.738287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:08.738248 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:09.520469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:09.519735 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:09.646471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:09.636857 4711 patch_prober.go:29] interesting pod/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:46:09.646471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:09.636924 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 20:46:09.664488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:09.663698 4711 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:46:09.664488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:09.663777 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 20:46:10.217860 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:10.217806 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:10.628949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:10.628904 4711 patch_prober.go:29] interesting pod/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:46:10.629136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:10.628976 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 03 20:46:10.660791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:10.660736 4711 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:46:10.660935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:10.660797 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 20:46:10.899786 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:10.899693 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:11.536522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:11.536493 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:46:11.603949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:11.603913 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:11.635780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:11.634969 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:46:11.636012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:11.635986 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:46:11.636911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:11.636868 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:46:11.660662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:11.660628 4711 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:46:11.660842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:11.660689 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 20:46:12.315711 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:12.315653 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:12.660737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:12.660655 4711 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:46:12.660737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:12.660704 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 20:46:13.009660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:13.009618 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:13.661266 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:13.661232 4711 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:46:13.661785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:13.661301 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 20:46:13.687466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:13.685315 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:14.400163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:14.400126 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:14.660664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:14.660586 4711 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:46:14.660664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:14.660639 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 03 20:46:15.098743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:15.098691 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:15.665925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:15.665890 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:15.666386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:15.666368 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:15.667162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:15.667143 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:46:15.858922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:15.858881 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:16.588052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:16.588018 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:17.340339 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:17.334356 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:17.533210 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:46:17.533168 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:46:18.132250 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:18.132210 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:19.337464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:19.337415 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:20.321105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:20.321067 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:21.042119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:21.042083 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:21.908294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:21.908250 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:22.705284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:22.705235 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:23.595616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:23.593130 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:24.228405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:24.228371 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:24.930981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:24.930946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:25.606420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:25.606385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:26.300312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:26.300272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:27.062868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:27.062816 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:27.827837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:27.827801 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:28.618605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:28.618554 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:29.321861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:29.321818 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:30.097689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:30.097642 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:30.753729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:30.753689 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:31.481587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:31.481549 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:32.380944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:32.380893 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:33.097970 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:33.097927 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:33.927051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:33.926961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:34.724754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:34.724701 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:34.854609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:34.854576 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s"] Nov 03 20:46:34.854857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:34.854829 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" containerID="cri-o://373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e" gracePeriod=30 Nov 03 20:46:34.859648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:34.859620 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv"] Nov 03 20:46:34.859883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:34.859854 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" containerID="cri-o://31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa" gracePeriod=30 Nov 03 20:46:35.512718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:35.512677 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:36.291843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:36.291803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:37.109627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:37.109590 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:37.436540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:37.436428 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:46:37.437022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:37.436997 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:46:37.440013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:37.439989 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:46:37.440198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:37.440180 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:46:37.995485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:37.995419 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:38.698838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:38.698793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:39.488522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:39.488485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:40.163906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.163874 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:46:40.214217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.214147 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:40.235161 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.235135 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:46:40.260406 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260369 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-token\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-token\") pod \"08bf17d6-5203-4647-927f-b5abc73ad8e9\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " Nov 03 20:46:40.260620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260434 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-socket\") pod \"08bf17d6-5203-4647-927f-b5abc73ad8e9\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " Nov 03 20:46:40.260620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260515 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-certs\") pod \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " Nov 03 20:46:40.260620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260571 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-envoy\") pod \"08bf17d6-5203-4647-927f-b5abc73ad8e9\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " Nov 03 20:46:40.260831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260715 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istiod-ca-cert\") pod \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " Nov 03 20:46:40.260831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260795 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08bf17d6-5203-4647-927f-b5abc73ad8e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf17d6-5203-4647-927f-b5abc73ad8e9-istiod-ca-cert\") pod \"08bf17d6-5203-4647-927f-b5abc73ad8e9\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " Nov 03 20:46:40.260831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260827 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-kube-api-access-rqtm5\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-kube-api-access-rqtm5\") pod \"08bf17d6-5203-4647-927f-b5abc73ad8e9\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " Nov 03 20:46:40.260939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260901 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-envoy\") pod \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " Nov 03 20:46:40.260939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260934 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-socket\") pod \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " Nov 03 20:46:40.261007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.260965 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-podinfo\") pod \"08bf17d6-5203-4647-927f-b5abc73ad8e9\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " Nov 03 20:46:40.261046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.261002 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-podinfo\") pod \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " Nov 03 20:46:40.261097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.261081 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-kube-api-access-r2zm2\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-kube-api-access-r2zm2\") pod \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " Nov 03 20:46:40.261145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.261130 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-data\") pod \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " Nov 03 20:46:40.261181 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.261162 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-credential-socket\") pod \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " Nov 03 20:46:40.261231 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.261218 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-data\") pod \"08bf17d6-5203-4647-927f-b5abc73ad8e9\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " Nov 03 20:46:40.261274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.261261 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-token\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-token\") pod \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\" (UID: \"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20\") " Nov 03 20:46:40.261306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.261294 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-credential-socket\") pod \"08bf17d6-5203-4647-927f-b5abc73ad8e9\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " Nov 03 20:46:40.261534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.261516 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-certs\") pod \"08bf17d6-5203-4647-927f-b5abc73ad8e9\" (UID: \"08bf17d6-5203-4647-927f-b5abc73ad8e9\") " Nov 03 20:46:40.262865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.262837 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-socket" pod "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" (UID: "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.262985 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.262960 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-socket" pod "08bf17d6-5203-4647-927f-b5abc73ad8e9" (UID: "08bf17d6-5203-4647-927f-b5abc73ad8e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.263091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.263069 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-data" pod "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" (UID: "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.263275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.262991 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istiod-ca-cert" pod "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" (UID: "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:40.263488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.263428 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-certs" pod "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" (UID: "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.263625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.263603 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bf17d6-5203-4647-927f-b5abc73ad8e9-istiod-ca-cert" pod "08bf17d6-5203-4647-927f-b5abc73ad8e9" (UID: "08bf17d6-5203-4647-927f-b5abc73ad8e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:40.263697 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.263685 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-data" pod "08bf17d6-5203-4647-927f-b5abc73ad8e9" (UID: "08bf17d6-5203-4647-927f-b5abc73ad8e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.263757 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.263704 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-certs" pod "08bf17d6-5203-4647-927f-b5abc73ad8e9" (UID: "08bf17d6-5203-4647-927f-b5abc73ad8e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.263873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.263861 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-credential-socket" pod "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" (UID: "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.263942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.263921 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-credential-socket" pod "08bf17d6-5203-4647-927f-b5abc73ad8e9" (UID: "08bf17d6-5203-4647-927f-b5abc73ad8e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.264518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.264492 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-token" pod "08bf17d6-5203-4647-927f-b5abc73ad8e9" (UID: "08bf17d6-5203-4647-927f-b5abc73ad8e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:40.266809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.266784 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-podinfo" pod "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" (UID: "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:46:40.266809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.266791 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-envoy" pod "08bf17d6-5203-4647-927f-b5abc73ad8e9" (UID: "08bf17d6-5203-4647-927f-b5abc73ad8e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.266809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.266803 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-podinfo" pod "08bf17d6-5203-4647-927f-b5abc73ad8e9" (UID: "08bf17d6-5203-4647-927f-b5abc73ad8e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:46:40.266998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.266934 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-token" pod "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" (UID: "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:40.266998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.266963 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-kube-api-access-rqtm5" pod "08bf17d6-5203-4647-927f-b5abc73ad8e9" (UID: "08bf17d6-5203-4647-927f-b5abc73ad8e9"). InnerVolumeSpecName "kube-api-access-rqtm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:40.267101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.267058 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-envoy" pod "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" (UID: "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:40.267975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.267951 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-kube-api-access-r2zm2" pod "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" (UID: "0a90ed4c-7a79-4d23-a35f-8e749c9c7f20"). InnerVolumeSpecName "kube-api-access-r2zm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:40.282271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.282239 4711 generic.go:356] "Generic (PLEG): container finished" podID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerID="373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e" exitCode=0 Nov 03 20:46:40.282496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.282286 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" Nov 03 20:46:40.282496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.282364 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" event={"ID":"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20","Type":"ContainerDied","Data":"373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e"} Nov 03 20:46:40.282496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.282423 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s" event={"ID":"0a90ed4c-7a79-4d23-a35f-8e749c9c7f20","Type":"ContainerDied","Data":"7aa51276f1551e6b291d1787fdb5748fe4eb98cae95b7259518ac8c5fe5775e9"} Nov 03 20:46:40.282496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.282461 4711 scope.go:122] "RemoveContainer" containerID="373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e" Nov 03 20:46:40.284685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.284656 4711 generic.go:356] "Generic (PLEG): container finished" podID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerID="31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa" exitCode=0 Nov 03 20:46:40.284812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.284757 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" Nov 03 20:46:40.284812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.284756 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" event={"ID":"08bf17d6-5203-4647-927f-b5abc73ad8e9","Type":"ContainerDied","Data":"31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa"} Nov 03 20:46:40.284812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.284811 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv" event={"ID":"08bf17d6-5203-4647-927f-b5abc73ad8e9","Type":"ContainerDied","Data":"67ade914644eb6993ea70e5a0160e137d05d5c8a023583d1f237be27d305f9ee"} Nov 03 20:46:40.305708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.305674 4711 scope.go:122] "RemoveContainer" containerID="373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e" Nov 03 20:46:40.306117 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:46:40.306087 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e\": container with ID starting with 373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e not found: ID does not exist" containerID="373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e" Nov 03 20:46:40.306254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.306132 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e"} err="failed to get container status \"373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e\": rpc error: code = NotFound desc = could not find container \"373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e\": container with ID starting with 373db7101cbb9f79d2c5ec2571426f3217e0c8c557abdab2f0c45e216570162e not found: ID does not exist" Nov 03 20:46:40.306254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.306157 4711 scope.go:122] "RemoveContainer" containerID="31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa" Nov 03 20:46:40.319254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.319219 4711 scope.go:122] "RemoveContainer" containerID="31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa" Nov 03 20:46:40.319713 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:46:40.319690 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa\": container with ID starting with 31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa not found: ID does not exist" containerID="31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa" Nov 03 20:46:40.319804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.319722 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa"} err="failed to get container status \"31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa\": rpc error: code = NotFound desc = could not find container \"31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa\": container with ID starting with 31ca92f90e215015d2f368511a35f02c55338525717d9e7bb5e42076d5f697fa not found: ID does not exist" Nov 03 20:46:40.331561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.331523 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv"] Nov 03 20:46:40.337097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.337067 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-72ddv"] Nov 03 20:46:40.347925 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.347872 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s"] Nov 03 20:46:40.352574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.352548 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-xcm8s"] Nov 03 20:46:40.364316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364275 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364316 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08bf17d6-5203-4647-927f-b5abc73ad8e9-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364327 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqtm5\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-kube-api-access-rqtm5\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364338 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364347 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364355 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364363 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364373 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zm2\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-kube-api-access-r2zm2\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364381 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364389 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364396 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364406 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364414 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364422 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364430 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364439 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364462 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.364607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.364470 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08bf17d6-5203-4647-927f-b5abc73ad8e9-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:40.682784 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.682755 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" path="/var/lib/kubelet/pods/08bf17d6-5203-4647-927f-b5abc73ad8e9/volumes" Nov 03 20:46:40.683257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:40.683242 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" path="/var/lib/kubelet/pods/0a90ed4c-7a79-4d23-a35f-8e749c9c7f20/volumes" Nov 03 20:46:41.005641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:41.005602 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:41.754471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:41.726346 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:42.432118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:42.432075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:42.618974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:42.618940 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4"] Nov 03 20:46:42.619177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:42.619153 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" containerID="cri-o://8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23" gracePeriod=30 Nov 03 20:46:43.220423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:43.220385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:44.061594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:44.061551 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:44.715408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:44.715360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:45.733684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:45.733643 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:45.879465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:45.878553 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:46.513411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:46.513326 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:46.727062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:46.727022 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:47.487416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:47.487378 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:47.608052 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:46:47.608005 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:46:47.889223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:47.889147 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:47.983359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:47.983334 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:46:48.043484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.031075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:48.082471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.070654 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-data\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.139482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.129618 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-data" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:48.139482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.135966 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-kube-api-access-pt26z\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-kube-api-access-pt26z\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.161468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.148595 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1703639c-2138-475d-aefd-275fef6f84f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1703639c-2138-475d-aefd-275fef6f84f6-istiod-ca-cert\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.161468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.148768 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.161468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.148849 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1703639c-2138-475d-aefd-275fef6f84f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1703639c-2138-475d-aefd-275fef6f84f6-istio-podinfo\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.161468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.148962 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-credential-socket\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.161468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.149051 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-socket\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.161468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.149100 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.161468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.149195 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-certs\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.161468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.150069 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:48.161468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.159520 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-socket" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:48.173468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.167533 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-credential-socket" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:48.263669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.202894 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1703639c-2138-475d-aefd-275fef6f84f6-istiod-ca-cert" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:48.263669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.258667 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:48.263669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.258700 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:48.263669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.258716 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1703639c-2138-475d-aefd-275fef6f84f6-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:48.283632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.283591 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-certs" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:48.326311 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.326083 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-kube-api-access-pt26z" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "kube-api-access-pt26z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.368617 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.368779 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1703639c-2138-475d-aefd-275fef6f84f6-istio-podinfo" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.370060 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.372780 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.373193 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy\") pod \"1703639c-2138-475d-aefd-275fef6f84f6\" (UID: \"1703639c-2138-475d-aefd-275fef6f84f6\") " Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.374424 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.374470 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt26z\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-kube-api-access-pt26z\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.374488 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1703639c-2138-475d-aefd-275fef6f84f6-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:46:48.374554 4711 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/1703639c-2138-475d-aefd-275fef6f84f6/volumes/kubernetes.io~projected/istio-token Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:46:48.374574 4711 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/1703639c-2138-475d-aefd-275fef6f84f6/volumes/kubernetes.io~empty-dir/istio-envoy Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.374588 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:46:48.377402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.374618 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token" pod "1703639c-2138-475d-aefd-275fef6f84f6" (UID: "1703639c-2138-475d-aefd-275fef6f84f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:48.383505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.383074 4711 generic.go:356] "Generic (PLEG): container finished" podID="1703639c-2138-475d-aefd-275fef6f84f6" containerID="8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23" exitCode=0 Nov 03 20:46:48.383505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.383293 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" event={"ID":"1703639c-2138-475d-aefd-275fef6f84f6","Type":"ContainerDied","Data":"8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23"} Nov 03 20:46:48.383505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.383362 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" event={"ID":"1703639c-2138-475d-aefd-275fef6f84f6","Type":"ContainerDied","Data":"e49d4eef9457a5c8880aa4577cad0c7ada461fd8b50f44b9a40fac23dfb78315"} Nov 03 20:46:48.383505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.383389 4711 scope.go:122] "RemoveContainer" containerID="8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23" Nov 03 20:46:48.383842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.383558 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4" Nov 03 20:46:48.409647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.409542 4711 scope.go:122] "RemoveContainer" containerID="cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff" Nov 03 20:46:48.423547 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.423524 4711 scope.go:122] "RemoveContainer" containerID="8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23" Nov 03 20:46:48.423934 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:46:48.423830 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23\": container with ID starting with 8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23 not found: ID does not exist" containerID="8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23" Nov 03 20:46:48.423934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.423872 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23"} err="failed to get container status \"8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23\": rpc error: code = NotFound desc = could not find container \"8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23\": container with ID starting with 8da2ef797f3262bbdc983796899bcff2cfbbf106a742c858c90a491b1bda5f23 not found: ID does not exist" Nov 03 20:46:48.423934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.423892 4711 scope.go:122] "RemoveContainer" containerID="cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff" Nov 03 20:46:48.424362 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:46:48.424316 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff\": container with ID starting with cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff not found: ID does not exist" containerID="cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff" Nov 03 20:46:48.424362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.424339 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff"} err="failed to get container status \"cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff\": rpc error: code = NotFound desc = could not find container \"cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff\": container with ID starting with cd4816442b08cf24927249f17b644710e3d1667857b3358a90a1d6cb4d3fbbff not found: ID does not exist" Nov 03 20:46:48.425037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.425024 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4"] Nov 03 20:46:48.431892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.431870 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-2gzr4"] Nov 03 20:46:48.476743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.476707 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1703639c-2138-475d-aefd-275fef6f84f6-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:48.476743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.476741 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1703639c-2138-475d-aefd-275fef6f84f6-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:48.520609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.520576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:48.712468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.705747 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:48.814474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:48.813518 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1703639c-2138-475d-aefd-275fef6f84f6" path="/var/lib/kubelet/pods/1703639c-2138-475d-aefd-275fef6f84f6/volumes" Nov 03 20:46:49.472217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:49.472168 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:50.239580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:50.224957 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:50.898934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:50.898894 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:51.716919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:51.716885 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:52.411600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:52.411548 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:53.072245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:53.072210 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:53.727970 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:53.727919 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:54.755612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:54.755580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:54.905076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:54.905017 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:55.216113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:55.216011 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:55.512956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:55.512919 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:55.652047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:55.651988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:55.798147 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:55.798059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:56.531466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:56.530074 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:57.417534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:57.417437 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:58.233281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:58.233235 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:58.784813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:58.784776 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:59.008954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:59.008912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:59.771697 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:59.771664 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:46:59.903285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:46:59.903235 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:00.041641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:00.041553 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:00.328518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:00.328361 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:00.723071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:00.722972 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:01.409510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:01.409470 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:01.543968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:01.543916 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:01.706462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:01.705476 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:02.415861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:02.415828 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:03.193704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:03.193671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:03.679624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:03.679586 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk"] Nov 03 20:47:03.680164 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:03.679850 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" podUID="40e08425-61fd-49bc-a532-f6b073382774" containerName="istio-proxy" containerID="cri-o://34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d" gracePeriod=30 Nov 03 20:47:03.893122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:03.893090 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:04.625009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:04.624963 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:05.444526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:05.444491 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:06.832123 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:06.831996 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:07.594748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:07.594696 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:08.303840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:08.303800 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:08.975487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:08.975430 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:47:08.991102 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:08.991076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:09.083271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.083207 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-data\") pod \"40e08425-61fd-49bc-a532-f6b073382774\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " Nov 03 20:47:09.083705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.083666 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-credential-socket\") pod \"40e08425-61fd-49bc-a532-f6b073382774\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " Nov 03 20:47:09.084219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.084182 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e08425-61fd-49bc-a532-f6b073382774-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e08425-61fd-49bc-a532-f6b073382774-istiod-ca-cert\") pod \"40e08425-61fd-49bc-a532-f6b073382774\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " Nov 03 20:47:09.084583 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.084566 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-kube-api-access-5fb95\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-kube-api-access-5fb95\") pod \"40e08425-61fd-49bc-a532-f6b073382774\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " Nov 03 20:47:09.086431 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.085560 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e08425-61fd-49bc-a532-f6b073382774-istiod-ca-cert" pod "40e08425-61fd-49bc-a532-f6b073382774" (UID: "40e08425-61fd-49bc-a532-f6b073382774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:09.087659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.086912 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-data" pod "40e08425-61fd-49bc-a532-f6b073382774" (UID: "40e08425-61fd-49bc-a532-f6b073382774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:09.088086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.088066 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40e08425-61fd-49bc-a532-f6b073382774-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e08425-61fd-49bc-a532-f6b073382774-istio-podinfo\") pod \"40e08425-61fd-49bc-a532-f6b073382774\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " Nov 03 20:47:09.088262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.088247 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-istio-token\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-istio-token\") pod \"40e08425-61fd-49bc-a532-f6b073382774\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " Nov 03 20:47:09.088911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.088392 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-credential-socket" pod "40e08425-61fd-49bc-a532-f6b073382774" (UID: "40e08425-61fd-49bc-a532-f6b073382774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:09.089119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.089089 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-socket\") pod \"40e08425-61fd-49bc-a532-f6b073382774\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " Nov 03 20:47:09.089996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.089461 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-socket" pod "40e08425-61fd-49bc-a532-f6b073382774" (UID: "40e08425-61fd-49bc-a532-f6b073382774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:09.090264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.090232 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-envoy\") pod \"40e08425-61fd-49bc-a532-f6b073382774\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " Nov 03 20:47:09.090455 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.090425 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-certs\") pod \"40e08425-61fd-49bc-a532-f6b073382774\" (UID: \"40e08425-61fd-49bc-a532-f6b073382774\") " Nov 03 20:47:09.091880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.091861 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.092028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.092013 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.092160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.092142 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.092285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.092273 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e08425-61fd-49bc-a532-f6b073382774-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.094257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.094233 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-certs" pod "40e08425-61fd-49bc-a532-f6b073382774" (UID: "40e08425-61fd-49bc-a532-f6b073382774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:09.143005 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.142926 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e08425-61fd-49bc-a532-f6b073382774-istio-podinfo" pod "40e08425-61fd-49bc-a532-f6b073382774" (UID: "40e08425-61fd-49bc-a532-f6b073382774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:47:09.143391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.143346 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-istio-token" pod "40e08425-61fd-49bc-a532-f6b073382774" (UID: "40e08425-61fd-49bc-a532-f6b073382774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:09.148898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.148549 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-envoy" pod "40e08425-61fd-49bc-a532-f6b073382774" (UID: "40e08425-61fd-49bc-a532-f6b073382774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:47:09.151065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.151034 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-kube-api-access-5fb95" pod "40e08425-61fd-49bc-a532-f6b073382774" (UID: "40e08425-61fd-49bc-a532-f6b073382774"). InnerVolumeSpecName "kube-api-access-5fb95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:09.194328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.194291 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.194328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.194330 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e08425-61fd-49bc-a532-f6b073382774-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.194612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.194346 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fb95\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-kube-api-access-5fb95\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.194612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.194359 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e08425-61fd-49bc-a532-f6b073382774-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.194612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.194369 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e08425-61fd-49bc-a532-f6b073382774-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.713645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.713615 4711 generic.go:356] "Generic (PLEG): container finished" podID="40e08425-61fd-49bc-a532-f6b073382774" containerID="34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d" exitCode=0 Nov 03 20:47:09.714043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.713669 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" Nov 03 20:47:09.714043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.713693 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" event={"ID":"40e08425-61fd-49bc-a532-f6b073382774","Type":"ContainerDied","Data":"34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d"} Nov 03 20:47:09.714043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.713747 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk" event={"ID":"40e08425-61fd-49bc-a532-f6b073382774","Type":"ContainerDied","Data":"871de2462ff35f384d132d7274d3c221d4f653868c2aa41585ff5a94d08f18e2"} Nov 03 20:47:09.714043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.713770 4711 scope.go:122] "RemoveContainer" containerID="34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d" Nov 03 20:47:09.717923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.717896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:09.784398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.770597 4711 scope.go:122] "RemoveContainer" containerID="34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d" Nov 03 20:47:09.784398 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:47:09.775593 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d\": container with ID starting with 34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d not found: ID does not exist" containerID="34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d" Nov 03 20:47:09.784398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.775638 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d"} err="failed to get container status \"34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d\": rpc error: code = NotFound desc = could not find container \"34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d\": container with ID starting with 34549c03090bf76ec86371900fc1e3681efdf11b0f5b8ae20c28854e746d768d not found: ID does not exist" Nov 03 20:47:09.790338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.790295 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk"] Nov 03 20:47:09.796629 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:09.796595 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-f9zzk"] Nov 03 20:47:10.423126 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:10.423090 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:10.682295 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:10.682225 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e08425-61fd-49bc-a532-f6b073382774" path="/var/lib/kubelet/pods/40e08425-61fd-49bc-a532-f6b073382774/volumes" Nov 03 20:47:11.150470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:11.148351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:11.794709 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:11.794672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:12.584920 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:12.584885 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:13.416028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:13.414610 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:14.185683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:14.185651 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:14.885097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:14.885054 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:15.515539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:15.515496 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:16.195338 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:16.195302 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:16.922902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:16.922857 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:17.534174 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:47:17.534136 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:47:17.605459 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:17.605411 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:18.273705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:18.273670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:19.013388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:19.013342 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:19.701906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:19.701867 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:20.435698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.435667 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5"] Nov 03 20:47:20.436523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436497 4711 cpu_manager.go:414] "containerMap: removing container" podUID="40e08425-61fd-49bc-a532-f6b073382774" containerName="istio-proxy" Nov 03 20:47:20.436523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436517 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e08425-61fd-49bc-a532-f6b073382774" containerName="istio-proxy" Nov 03 20:47:20.436733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436550 4711 cpu_manager.go:414] "containerMap: removing container" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" Nov 03 20:47:20.436733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436555 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" Nov 03 20:47:20.436733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436566 4711 cpu_manager.go:414] "containerMap: removing container" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" Nov 03 20:47:20.436733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436573 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" Nov 03 20:47:20.436733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436597 4711 cpu_manager.go:414] "containerMap: removing container" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" Nov 03 20:47:20.436733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436601 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" Nov 03 20:47:20.436733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436610 4711 cpu_manager.go:414] "containerMap: removing container" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" Nov 03 20:47:20.436733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436614 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" Nov 03 20:47:20.437155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436755 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" Nov 03 20:47:20.437155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436768 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="1703639c-2138-475d-aefd-275fef6f84f6" containerName="istio-proxy" Nov 03 20:47:20.437155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436785 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="08bf17d6-5203-4647-927f-b5abc73ad8e9" containerName="istio-proxy" Nov 03 20:47:20.437155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436794 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a90ed4c-7a79-4d23-a35f-8e749c9c7f20" containerName="istio-proxy" Nov 03 20:47:20.437155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.436804 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e08425-61fd-49bc-a532-f6b073382774" containerName="istio-proxy" Nov 03 20:47:20.442092 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.442069 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.450924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.450887 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bf7965-serving-cert\"" Nov 03 20:47:20.457278 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.457255 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5"] Nov 03 20:47:20.495962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.495928 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-metrics-certs\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.495962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.495966 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.496218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.496189 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8g48\" (UniqueName: \"kubernetes.io/projected/4aa7c6ac-d377-450b-820b-19fb08c44a55-kube-api-access-k8g48\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.496300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.496283 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa7c6ac-d377-450b-820b-19fb08c44a55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.517614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.517581 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:20.630544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.607081 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-metrics-certs\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.630910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.630869 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.631727 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.631702 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8g48\" (UniqueName: \"kubernetes.io/projected/4aa7c6ac-d377-450b-820b-19fb08c44a55-kube-api-access-k8g48\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.640580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.640547 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa7c6ac-d377-450b-820b-19fb08c44a55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.641002 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.639852 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-maas-api-tls\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.643433 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.643410 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa7c6ac-d377-450b-820b-19fb08c44a55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.648328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.648308 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8g48\" (UniqueName: \"kubernetes.io/projected/4aa7c6ac-d377-450b-820b-19fb08c44a55-kube-api-access-k8g48\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.649875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.649853 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-metrics-certs\") pod \"maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.656149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.656126 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:20.771859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.771831 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:20.786847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:20.786817 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:21.062496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:21.062432 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5"] Nov 03 20:47:21.062671 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:47:21.062646 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa7c6ac_d377_450b_820b_19fb08c44a55.slice/crio-d9471bdaa4b35db888077c892efa437eff5e6f6f4afdab1d030bf1836cf7d299 WatchSource:0}: Error finding container d9471bdaa4b35db888077c892efa437eff5e6f6f4afdab1d030bf1836cf7d299: Status 404 returned error can't find the container with id d9471bdaa4b35db888077c892efa437eff5e6f6f4afdab1d030bf1836cf7d299 Nov 03 20:47:21.471656 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:21.471531 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:21.892467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:21.877202 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" event={"ID":"4aa7c6ac-d377-450b-820b-19fb08c44a55","Type":"ContainerStarted","Data":"8402d2fce6d542a58342da8ee62c4f87e6f6adc6e49a4cb7571028499dd3cad4"} Nov 03 20:47:21.892467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:21.877271 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" event={"ID":"4aa7c6ac-d377-450b-820b-19fb08c44a55","Type":"ContainerStarted","Data":"d9471bdaa4b35db888077c892efa437eff5e6f6f4afdab1d030bf1836cf7d299"} Nov 03 20:47:21.892467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:21.878509 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:21.932470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:21.932251 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" podStartSLOduration=1.412998741 podStartE2EDuration="1.932230774s" totalImagesPullingTime="519.232033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:21.903831171 +0000 UTC m=+2745.353937229" watchObservedRunningTime="2026-11-03 20:47:21.932230774 +0000 UTC m=+2745.382336823" Nov 03 20:47:22.404195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:22.404158 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:23.096400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:23.096357 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:23.910350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:23.910315 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:24.613677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:24.613623 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:25.390276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:25.390241 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:26.125380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:26.125326 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:26.836471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:26.829468 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:27.496691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:27.496653 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:28.213567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:28.213517 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:28.890764 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:28.890725 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:28.897721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:28.897697 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:47:29.621848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:29.621810 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:30.423467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:30.423332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:31.197824 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:31.197787 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:31.723489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:31.714718 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:31.886052 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:31.885970 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:32.573454 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:32.573421 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:33.365411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:33.365374 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:34.077480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:34.077241 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:34.772255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:34.772200 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:35.394900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:35.394851 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:36.214333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:36.214238 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:36.939770 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:36.939717 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:37.591209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:37.591152 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:38.319208 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:38.319167 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:39.000708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:39.000671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:39.660254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:39.660037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:40.433753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:40.433631 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:41.140914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:41.130621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:41.794928 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:41.794892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:42.535973 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:42.535932 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:43.378810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:43.378774 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:44.204574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:44.204534 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:44.967692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:44.967643 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:45.693997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:45.693962 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:45.842473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:45.833130 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:46.378053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:46.378011 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:46.506705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:46.506671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:46.911469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:46.909641 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:47.389391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:47.389310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:47.522855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:47.522823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:47.538978 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:47:47.538899 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:47:48.107677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:48.107584 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:48.794916 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:48.794882 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:49.686890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:49.686834 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:50.515938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:50.515885 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:50.992858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:50.992771 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:51.134764 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:51.134728 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:51.729569 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:51.729530 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:52.219317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:52.219282 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:52.378982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:52.378947 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:52.627047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:52.627013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:53.323477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:53.323370 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:53.685616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:53.683748 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:47:54.413281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:54.413223 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:54.767106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:54.767058 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:54.916473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:54.901753 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.150523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:55.150413 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.414527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:55.411616 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.636679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:55.636639 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:55.812810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:55.812768 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:56.629274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:56.629233 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:57.767538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:57.767500 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:58.527826 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:58.527785 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:58.824539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:58.824463 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:59.350474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:59.348461 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:47:59.830333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:47:59.830295 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:00.017748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:00.017712 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:00.765220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:00.765183 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:01.617050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:01.617016 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:01.757408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:01.757370 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:01.895648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:01.895553 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:02.251510 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:02.251381 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:02.668423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:02.668384 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:02.814468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:02.805963 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:03.407353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:03.407315 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:04.118870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:04.118834 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:04.941484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:04.920397 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:05.885264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:05.885232 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:06.160624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:06.160541 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:06.290270 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:06.290235 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:06.869370 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:06.869339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:07.342657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:07.342491 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:07.486662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:07.486612 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:07.715401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:07.715312 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:08.826086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:08.826047 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:09.623229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:09.623190 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:10.353470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:10.347488 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:11.075500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:11.067405 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:11.798116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:11.798079 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:12.499425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:12.499387 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:13.322915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:13.322852 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:14.004135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:14.004088 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:14.660622 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:14.660585 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:15.320654 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:15.320621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:15.918316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:15.918278 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2"] Nov 03 20:48:15.923540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:15.923515 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:15.935036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:15.935010 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d7c7fd-serving-cert\"" Nov 03 20:48:15.937844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:15.937817 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2"] Nov 03 20:48:16.008292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.008251 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:16.028575 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.028539 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-metrics-certs\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.028761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.028609 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zck59\" (UniqueName: \"kubernetes.io/projected/b16636b8-cceb-41ef-af18-8052c9429822-kube-api-access-zck59\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.028761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.028730 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16636b8-cceb-41ef-af18-8052c9429822-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.028956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.028938 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.142470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.139688 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.142470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.139909 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-metrics-certs\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.142470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.140562 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zck59\" (UniqueName: \"kubernetes.io/projected/b16636b8-cceb-41ef-af18-8052c9429822-kube-api-access-zck59\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.142470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.141379 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16636b8-cceb-41ef-af18-8052c9429822-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.159551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.157631 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16636b8-cceb-41ef-af18-8052c9429822-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.183525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.167098 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-metrics-certs\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.183525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.181176 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:16.191481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.184373 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-maas-api-tls\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.210264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.199280 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zck59\" (UniqueName: \"kubernetes.io/projected/b16636b8-cceb-41ef-af18-8052c9429822-kube-api-access-zck59\") pod \"maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.254178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.254037 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:16.544775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.544738 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2"] Nov 03 20:48:16.546130 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:48:16.546101 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16636b8_cceb_41ef_af18_8052c9429822.slice/crio-381329f052f944d54baccdec8e36679c3359c4e9a11e6ff970cfc0100fb626c6 WatchSource:0}: Error finding container 381329f052f944d54baccdec8e36679c3359c4e9a11e6ff970cfc0100fb626c6: Status 404 returned error can't find the container with id 381329f052f944d54baccdec8e36679c3359c4e9a11e6ff970cfc0100fb626c6 Nov 03 20:48:16.793774 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:16.791164 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" event={"ID":"b16636b8-cceb-41ef-af18-8052c9429822","Type":"ContainerStarted","Data":"381329f052f944d54baccdec8e36679c3359c4e9a11e6ff970cfc0100fb626c6"} Nov 03 20:48:17.003952 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:17.003920 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:17.535248 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:48:17.535211 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:48:17.800811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:17.800722 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" event={"ID":"b16636b8-cceb-41ef-af18-8052c9429822","Type":"ContainerStarted","Data":"28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489"} Nov 03 20:48:17.800959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:17.800825 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:17.822773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:17.822725 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" podStartSLOduration=2.408271588 podStartE2EDuration="2.822711348s" totalImagesPullingTime="414.43976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:17.820760017 +0000 UTC m=+2801.270866061" watchObservedRunningTime="2026-11-03 20:48:17.822711348 +0000 UTC m=+2801.272817395" Nov 03 20:48:17.892093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:17.892052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:18.618708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:18.618671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:19.289613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:19.289575 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:20.064465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:20.063362 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:20.709398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:20.709368 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:21.405527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:21.405491 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:22.205477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:22.205390 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:23.090684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:23.090646 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:23.795746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:23.795700 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:23.884213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:23.884180 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:48:24.691261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:24.691228 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:25.506414 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:25.506381 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:26.219483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:26.216934 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:27.219609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:27.219579 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:27.940211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:27.940178 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:28.699456 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:28.699412 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:29.430692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.430658 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764"] Nov 03 20:48:29.437029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.437004 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.449296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.449261 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764"] Nov 03 20:48:29.451093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.451067 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4d63cd-serving-cert\"" Nov 03 20:48:29.506256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.506220 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:29.583951 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.583912 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zft4b\" (UniqueName: \"kubernetes.io/projected/c281bd1d-9536-4754-8f2c-75596c7f9ac6-kube-api-access-zft4b\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.584314 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.584292 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c281bd1d-9536-4754-8f2c-75596c7f9ac6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.589178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.584625 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-metrics-certs\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.589178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.588034 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.638855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.638820 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:29.690494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.690378 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-metrics-certs\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.690679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.690492 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.690749 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.690712 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zft4b\" (UniqueName: \"kubernetes.io/projected/c281bd1d-9536-4754-8f2c-75596c7f9ac6-kube-api-access-zft4b\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.690809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.690780 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c281bd1d-9536-4754-8f2c-75596c7f9ac6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.691490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.691464 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c281bd1d-9536-4754-8f2c-75596c7f9ac6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.693032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.693013 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-metrics-certs\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.693137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.693119 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-maas-api-tls\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.703411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.703384 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zft4b\" (UniqueName: \"kubernetes.io/projected/c281bd1d-9536-4754-8f2c-75596c7f9ac6-kube-api-access-zft4b\") pod \"maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:29.765906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:29.765878 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:30.148242 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:48:30.148206 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc281bd1d_9536_4754_8f2c_75596c7f9ac6.slice/crio-aea22079f3f718a22612e5463cb80a523c7e7e6a512fe55c2b87f495b2f2cda3 WatchSource:0}: Error finding container aea22079f3f718a22612e5463cb80a523c7e7e6a512fe55c2b87f495b2f2cda3: Status 404 returned error can't find the container with id aea22079f3f718a22612e5463cb80a523c7e7e6a512fe55c2b87f495b2f2cda3 Nov 03 20:48:30.148934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:30.148908 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764"] Nov 03 20:48:30.397427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:30.396727 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:30.943582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:30.943542 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" event={"ID":"c281bd1d-9536-4754-8f2c-75596c7f9ac6","Type":"ContainerStarted","Data":"17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718"} Nov 03 20:48:30.943991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:30.943608 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" event={"ID":"c281bd1d-9536-4754-8f2c-75596c7f9ac6","Type":"ContainerStarted","Data":"aea22079f3f718a22612e5463cb80a523c7e7e6a512fe55c2b87f495b2f2cda3"} Nov 03 20:48:30.943991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:30.943648 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:30.965913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:30.965856 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" podStartSLOduration=1.3728824 podStartE2EDuration="1.965836964s" totalImagesPullingTime="592.954564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:30.964553672 +0000 UTC m=+2814.414659719" watchObservedRunningTime="2026-11-03 20:48:30.965836964 +0000 UTC m=+2814.415943015" Nov 03 20:48:31.131024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:31.130983 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:31.827162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:31.827124 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:32.515922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:32.515883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:33.308773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:33.308726 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:34.026138 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:34.026103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:34.900145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:34.900098 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:36.127200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:36.127157 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:36.956902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:36.956872 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:48:37.115796 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:37.115763 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:37.644775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:37.644735 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:38.013981 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:38.013952 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:38.690080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:38.690031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:39.421230 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:39.421198 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:40.100779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:40.100740 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:40.749669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:40.749637 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:41.789873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:41.789668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:42.592777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:42.592709 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:43.287173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:43.287138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:44.015221 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:44.015180 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:44.673627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:44.673587 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:45.359389 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:45.359355 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:46.110552 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:46.110518 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:46.793840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:46.793801 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:47.028089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:47.028048 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:47.536666 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:48:47.536623 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:48:47.657476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:47.650992 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:47.922902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:47.922795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:48.573274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:48.573090 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:49.197343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:49.197310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:49.923910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:49.923871 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:50.707699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:50.707641 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:51.482952 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:51.482896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:52.317288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:52.317253 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:53.053877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:53.053836 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:53.731754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:53.731697 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:54.594000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:54.593967 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:55.418903 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:55.418862 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:56.280106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:56.280071 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:57.063273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.063238 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:57.199511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.199429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:57.513847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.513809 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:57.859758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.859671 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h"] Nov 03 20:48:57.865154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.865129 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:57.884559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.884528 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h"] Nov 03 20:48:57.938390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.938357 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-519a78-openshift-default-dockercfg-x7qmg\"" Nov 03 20:48:57.953463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.953417 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:57.964434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.964401 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-certs\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:57.964630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.964504 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-token\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:57.964630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.964603 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-podinfo\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:57.964733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.964645 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9k2\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-kube-api-access-dq9k2\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:57.964733 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.964711 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-credential-socket\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:57.964819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.964767 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-data\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:57.964967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.964952 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-socket\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:57.965093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.965077 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-envoy\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:57.965224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:57.965119 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istiod-ca-cert\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067035 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-envoy\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067079 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istiod-ca-cert\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067119 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-certs\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067137 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-token\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067308 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067262 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-podinfo\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067306 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9k2\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-kube-api-access-dq9k2\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067351 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-credential-socket\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067410 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-data\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067736 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-certs\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067797 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-credential-socket\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.067915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067884 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-socket\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.068000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.067978 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-data\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.068087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.068005 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istiod-ca-cert\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.068179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.068160 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-socket\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.069853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.069830 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-envoy\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.069960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.069874 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-podinfo\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.081366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.081340 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-token\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.081505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.081410 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9k2\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-kube-api-access-dq9k2\") pod \"e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.164524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.157035 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:58.198467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.186740 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:48:58.666188 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:48:58.666151 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bf29b7_c7b6_4200_b2f3_0bdbe95e29d9.slice/crio-f9c45c319fad731e45e18f7d02b6e0f5ab87e1191669733bdfc4b5255e98a403 WatchSource:0}: Error finding container f9c45c319fad731e45e18f7d02b6e0f5ab87e1191669733bdfc4b5255e98a403: Status 404 returned error can't find the container with id f9c45c319fad731e45e18f7d02b6e0f5ab87e1191669733bdfc4b5255e98a403 Nov 03 20:48:58.666899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.666721 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h"] Nov 03 20:48:58.669586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.669552 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:48:58.669683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.669620 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:48:58.669683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.669650 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:48:58.897665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:58.897625 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:59.031851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:59.031805 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:48:59.547974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:59.547938 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" event={"ID":"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9","Type":"ContainerStarted","Data":"206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c"} Nov 03 20:48:59.548151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:59.548002 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" event={"ID":"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9","Type":"ContainerStarted","Data":"f9c45c319fad731e45e18f7d02b6e0f5ab87e1191669733bdfc4b5255e98a403"} Nov 03 20:48:59.581529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:59.581428 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" podStartSLOduration=2.581408809 podStartE2EDuration="2.581408809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:59.578497746 +0000 UTC m=+2843.028603794" watchObservedRunningTime="2026-11-03 20:48:59.581408809 +0000 UTC m=+2843.031514855" Nov 03 20:48:59.727699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:48:59.727659 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:00.064177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:00.064141 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:00.187725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:00.187690 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:49:00.195303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:00.195279 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:49:00.207858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:00.207828 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:00.559874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:00.559841 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:49:00.560992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:00.560971 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:49:00.637012 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:00.636978 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:01.391468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:01.388209 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:01.752219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:01.752130 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:01.952481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:01.940550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:02.397477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:02.379739 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:03.126382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.126343 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:03.195921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.195874 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq"] Nov 03 20:49:03.201827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.201802 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:03.205020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.204996 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq"] Nov 03 20:49:03.238394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.223266 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-519a78-dockercfg-qxd5z\"" Nov 03 20:49:03.242163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.242141 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-519a78\"" Nov 03 20:49:03.262101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.262068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:03.403604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.403205 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4kx\" (UniqueName: \"kubernetes.io/projected/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-kube-api-access-vg4kx\") pod \"payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq\" (UID: \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\") " pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:03.403604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.403359 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq\" (UID: \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\") " pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:03.446409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.446372 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:03.508828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.508787 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4kx\" (UniqueName: \"kubernetes.io/projected/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-kube-api-access-vg4kx\") pod \"payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq\" (UID: \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\") " pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:03.509157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.509136 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq\" (UID: \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\") " pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:03.510260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.510234 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq\" (UID: \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\") " pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:03.582542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.579798 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4kx\" (UniqueName: \"kubernetes.io/projected/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-kube-api-access-vg4kx\") pod \"payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq\" (UID: \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\") " pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:03.849400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:03.849362 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:04.136657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:04.136623 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq"] Nov 03 20:49:04.138531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:04.138501 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:04.633409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:04.633378 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" event={"ID":"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64","Type":"ContainerStarted","Data":"2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5"} Nov 03 20:49:04.633836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:04.633429 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" event={"ID":"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64","Type":"ContainerStarted","Data":"a5738fdbe07352e2073766532c03b0cde1a49d6a24aa0c60129220a866bfe3a5"} Nov 03 20:49:04.633836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:04.633495 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:04.660004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:04.659952 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" podStartSLOduration=1.6599392069999999 podStartE2EDuration="1.659939207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:04.656773789 +0000 UTC m=+2848.106879835" watchObservedRunningTime="2026-11-03 20:49:04.659939207 +0000 UTC m=+2848.110045252" Nov 03 20:49:04.919930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:04.919841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:05.691055 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:05.691019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:06.405169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:06.405134 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:07.109848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:07.109813 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:07.794172 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:07.794136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:08.576600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:08.576569 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:09.409390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:09.409341 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:10.130038 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:10.128046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:10.800000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:10.799957 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:11.516650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:11.516610 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:12.212822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:12.212782 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:12.864738 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:12.864702 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:13.713730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:13.713691 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:14.387696 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:14.387661 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:15.193883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:15.193825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:15.644971 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:15.644934 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:49:15.908707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:15.908624 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:16.591419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:16.591382 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:17.361485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:17.357052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:17.533859 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:49:17.533825 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:49:18.058574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:18.058539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:18.881325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:18.881290 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:19.614647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:19.614595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:20.342513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:20.342471 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:20.478564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:20.478523 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:20.910435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:20.910397 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:21.422900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:21.422858 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:21.562101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:21.562065 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:21.698559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:21.698459 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:22.329019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:22.328984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:23.050478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.040637 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:23.621196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.621127 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf"] Nov 03 20:49:23.627474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.627431 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.632492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.631832 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-519a78-serving-cert\"" Nov 03 20:49:23.635533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.635506 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf"] Nov 03 20:49:23.690822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.690791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:23.824468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.798519 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-metrics-certs\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.824771 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.824735 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-maas-api-tls\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.825059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.825033 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40b733-4aaf-445c-a516-53810817296c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.825198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.825180 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-692q7\" (UniqueName: \"kubernetes.io/projected/cc40b733-4aaf-445c-a516-53810817296c-kube-api-access-692q7\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.841481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.837347 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:23.926708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.926630 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40b733-4aaf-445c-a516-53810817296c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.926708 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.926672 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-692q7\" (UniqueName: \"kubernetes.io/projected/cc40b733-4aaf-445c-a516-53810817296c-kube-api-access-692q7\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.927006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.926987 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-metrics-certs\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.927061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.927025 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-maas-api-tls\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.928011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.927750 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40b733-4aaf-445c-a516-53810817296c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.940113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.940080 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-692q7\" (UniqueName: \"kubernetes.io/projected/cc40b733-4aaf-445c-a516-53810817296c-kube-api-access-692q7\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.946249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.946227 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-metrics-certs\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.947303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.947281 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-maas-api-tls\") pod \"maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:23.990975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:23.990943 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:24.247255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:24.247181 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:24.721388 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:24.721356 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf"] Nov 03 20:49:24.721596 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:49:24.721515 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc40b733_4aaf_445c_a516_53810817296c.slice/crio-c28bee3f1569e5a117e1dffa08ef7e9b45cd4ef4e424f3c9b6b82f907087804b WatchSource:0}: Error finding container c28bee3f1569e5a117e1dffa08ef7e9b45cd4ef4e424f3c9b6b82f907087804b: Status 404 returned error can't find the container with id c28bee3f1569e5a117e1dffa08ef7e9b45cd4ef4e424f3c9b6b82f907087804b Nov 03 20:49:24.813111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:24.813080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:24.907817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:24.907778 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" event={"ID":"cc40b733-4aaf-445c-a516-53810817296c","Type":"ContainerStarted","Data":"c28bee3f1569e5a117e1dffa08ef7e9b45cd4ef4e424f3c9b6b82f907087804b"} Nov 03 20:49:25.932101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:25.932061 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" event={"ID":"cc40b733-4aaf-445c-a516-53810817296c","Type":"ContainerStarted","Data":"c581e51a29107feaa6db5980ba4d057424b13c3be040b3cd53474657b50abbf6"} Nov 03 20:49:25.932662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:25.932125 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:25.937089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:25.937058 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:25.955524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:25.955410 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" podStartSLOduration=2.466940509 podStartE2EDuration="2.955397226s" totalImagesPullingTime="488.456717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:25.953181778 +0000 UTC m=+2869.403287807" watchObservedRunningTime="2026-11-03 20:49:25.955397226 +0000 UTC m=+2869.405503271" Nov 03 20:49:26.687200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:26.687165 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:27.689524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:27.689491 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:28.358479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:28.358410 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:29.125026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:29.124992 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:29.840819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:29.840782 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:30.592493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:30.592425 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:31.337333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:31.337304 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:31.965959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:31.965931 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:49:32.117377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:32.117334 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:32.799729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:32.799696 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:33.492577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:33.492525 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:34.321640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:34.321602 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:34.999384 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:34.999348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:35.814646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:35.814555 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:36.503797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:36.501655 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:37.326855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:37.326813 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:38.012665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:38.012624 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:38.696919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:38.696874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:39.403768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:39.403728 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:40.217347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:40.217311 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:40.891734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:40.891701 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:41.724693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:41.724642 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:42.414640 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:42.414605 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:43.101746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:43.101712 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:43.821048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:43.821014 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:44.586089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:44.586052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:45.329523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:45.329485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:45.999792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:45.999749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:46.701611 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:46.701511 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:47.305286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:47.305253 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:47.454705 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:47.454668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:47.565167 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:49:47.564965 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:49:47.586722 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:47.586690 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:48.227501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:48.227465 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:49.139724 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:49.139686 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:49.913453 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:49.913414 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:50.595903 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:50.595869 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:51.297237 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:51.297201 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:52.112854 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:52.112821 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:52.643563 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:52.643511 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:52.814931 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:52.814897 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:53.546179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:53.546144 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:54.059220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:54.059165 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:54.336483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:54.336068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:55.013020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:55.012980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:55.709607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:55.709566 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:56.496413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:56.496368 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:57.355844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:57.355803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:58.146238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:58.146183 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:58.867322 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:58.867285 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:58.999856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:58.999823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:49:59.616273 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:49:59.616215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:00.113573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:00.113533 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:00.244682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:00.244646 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:00.418521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:00.417395 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:01.041942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:01.041905 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:01.896712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:01.896647 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:02.605468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:02.605067 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:02.793068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:02.793034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:03.084546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:03.084507 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:03.896605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:03.896511 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:04.893049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:04.893012 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:05.600480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:05.599971 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:06.292515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:06.292479 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:06.697219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:06.697123 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:07.222048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:07.222013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:07.899000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:07.898965 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:08.722057 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:08.722017 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:09.505952 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:09.505914 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:10.273856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:10.255626 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:10.922808 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:10.922769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:11.596256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:11.596218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:12.323218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:12.323171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:13.192076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:13.191940 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:13.610915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:13.610886 4711 generic.go:356] "Generic (PLEG): container finished" podID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerID="206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c" exitCode=0 Nov 03 20:50:13.611411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:13.610959 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" event={"ID":"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9","Type":"ContainerDied","Data":"206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c"} Nov 03 20:50:13.611613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:13.611582 4711 scope.go:122] "RemoveContainer" containerID="206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c" Nov 03 20:50:13.613763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:13.613732 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:50:13.613879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:13.613819 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:50:13.613879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:13.613868 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:50:13.997169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:13.997077 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:14.622244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:14.622208 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" event={"ID":"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9","Type":"ContainerStarted","Data":"f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16"} Nov 03 20:50:14.896192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:14.896095 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:15.189903 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:15.189812 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:50:15.190271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:15.190246 4711 patch_prober.go:29] interesting pod/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:50:15.192576 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:15.192528 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 20:50:15.559853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:15.559806 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:15.560041 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:15.559822 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:50:16.191280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:16.191247 4711 patch_prober.go:29] interesting pod/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:50:16.191827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:16.191304 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 20:50:16.295004 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:16.294966 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:17.008806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:17.008773 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:17.193546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:17.190872 4711 patch_prober.go:29] interesting pod/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:50:17.193546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:17.190934 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 20:50:17.534823 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:50:17.534786 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:50:17.708680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:17.708645 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:18.190469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:18.189808 4711 patch_prober.go:29] interesting pod/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:50:18.190469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:18.189870 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 20:50:18.421735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:18.421695 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:19.096111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:19.096074 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:19.215881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:19.215833 4711 patch_prober.go:29] interesting pod/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Nov 03 20:50:19.231152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:19.231102 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Nov 03 20:50:19.823059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:19.823016 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:20.195120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:20.195035 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:50:20.197168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:20.197140 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:50:20.587421 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:20.587379 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:21.259532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:21.259499 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:21.999385 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:21.999348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:22.652194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:22.652121 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:23.316540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:23.316502 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:24.045471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:24.045361 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:24.663532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:24.663493 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:25.398995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:25.398961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:26.200613 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:26.200576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:26.920495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:26.917464 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:27.715734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:27.714285 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:28.421437 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:28.421294 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:29.104032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:29.103987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:29.933101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:29.933057 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:30.713989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:30.713959 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:31.537471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:31.537283 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:32.219116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:32.219072 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:32.903076 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:32.903039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:33.713418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:33.713382 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:34.495137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:34.495096 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:35.156871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:35.156834 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:35.917345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:35.917308 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:36.601136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:36.601097 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:37.367043 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:37.367001 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:38.116935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:38.116895 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:38.795391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:38.795329 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:39.461139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:39.461103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:40.291387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:40.291330 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:40.862743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:40.862707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:41.032478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:41.031301 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:41.708000 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:41.707964 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:42.545071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:42.545036 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:43.237623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:43.237582 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:44.236959 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:44.236921 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:45.586495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:45.586402 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:45.711661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:45.710843 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:50:46.583669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:46.583633 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:46.854099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:46.854007 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q"] Nov 03 20:50:46.854563 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:46.854209 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" containerID="cri-o://2160c3c559c1b62b57b8a5b118ca39b4575d3308f958447257be04a1bf385306" gracePeriod=30 Nov 03 20:50:47.355965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:47.355924 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:47.534357 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:50:47.534320 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:50:48.111697 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:48.111603 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:49.001559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:49.001524 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:49.712806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:49.712769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:50.617131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:50.617099 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:51.302878 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:51.302846 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:52.199108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.199058 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:52.229028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.229003 4711 generic.go:356] "Generic (PLEG): container finished" podID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerID="2160c3c559c1b62b57b8a5b118ca39b4575d3308f958447257be04a1bf385306" exitCode=0 Nov 03 20:50:52.229394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.229369 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" event={"ID":"7c8f34d6-5147-405a-8c9f-ea912c4b6b34","Type":"ContainerDied","Data":"2160c3c559c1b62b57b8a5b118ca39b4575d3308f958447257be04a1bf385306"} Nov 03 20:50:52.229617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.229601 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" event={"ID":"7c8f34d6-5147-405a-8c9f-ea912c4b6b34","Type":"ContainerDied","Data":"5da5e93a41774391ba45c1806ce4b8421439489be9b77f79b137f6c3a2ec791e"} Nov 03 20:50:52.229712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.229702 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5da5e93a41774391ba45c1806ce4b8421439489be9b77f79b137f6c3a2ec791e" Nov 03 20:50:52.240987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.240964 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:50:52.300175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.300136 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-podinfo\") pod \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " Nov 03 20:50:52.300383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.300207 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-envoy\") pod \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " Nov 03 20:50:52.300383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.300276 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-token\") pod \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " Nov 03 20:50:52.300635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.300425 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-certs\") pod \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " Nov 03 20:50:52.300635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.300478 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-data\") pod \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " Nov 03 20:50:52.300635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.300510 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-credential-socket\") pod \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " Nov 03 20:50:52.300635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.300594 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-kube-api-access-9d4fb\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-kube-api-access-9d4fb\") pod \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " Nov 03 20:50:52.300892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.300838 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istiod-ca-cert\") pod \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " Nov 03 20:50:52.300976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.300937 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-socket\") pod \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\" (UID: \"7c8f34d6-5147-405a-8c9f-ea912c4b6b34\") " Nov 03 20:50:52.302383 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.302352 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-socket" pod "7c8f34d6-5147-405a-8c9f-ea912c4b6b34" (UID: "7c8f34d6-5147-405a-8c9f-ea912c4b6b34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:50:52.302615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.302589 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-certs" pod "7c8f34d6-5147-405a-8c9f-ea912c4b6b34" (UID: "7c8f34d6-5147-405a-8c9f-ea912c4b6b34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:50:52.302821 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.302801 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-data" pod "7c8f34d6-5147-405a-8c9f-ea912c4b6b34" (UID: "7c8f34d6-5147-405a-8c9f-ea912c4b6b34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:50:52.302911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.302860 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-credential-socket" pod "7c8f34d6-5147-405a-8c9f-ea912c4b6b34" (UID: "7c8f34d6-5147-405a-8c9f-ea912c4b6b34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:50:52.303276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.303029 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istiod-ca-cert" pod "7c8f34d6-5147-405a-8c9f-ea912c4b6b34" (UID: "7c8f34d6-5147-405a-8c9f-ea912c4b6b34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:50:52.306723 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.306695 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-kube-api-access-9d4fb" pod "7c8f34d6-5147-405a-8c9f-ea912c4b6b34" (UID: "7c8f34d6-5147-405a-8c9f-ea912c4b6b34"). InnerVolumeSpecName "kube-api-access-9d4fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:50:52.310175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.310141 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-token" pod "7c8f34d6-5147-405a-8c9f-ea912c4b6b34" (UID: "7c8f34d6-5147-405a-8c9f-ea912c4b6b34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:50:52.312998 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.312953 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-envoy" pod "7c8f34d6-5147-405a-8c9f-ea912c4b6b34" (UID: "7c8f34d6-5147-405a-8c9f-ea912c4b6b34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:50:52.316333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.316296 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-podinfo" pod "7c8f34d6-5147-405a-8c9f-ea912c4b6b34" (UID: "7c8f34d6-5147-405a-8c9f-ea912c4b6b34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:50:52.404048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404015 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:52.404048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404042 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:52.404048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404051 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:52.404367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404059 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:52.404367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404068 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:52.404367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404076 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:52.404367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404086 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:52.404367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404099 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:52.404367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404112 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4fb\" (UniqueName: \"kubernetes.io/projected/7c8f34d6-5147-405a-8c9f-ea912c4b6b34-kube-api-access-9d4fb\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:52.404887 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.404861 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:52.898529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:52.898494 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:53.105605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:53.104207 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:53.237150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:53.237119 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q" Nov 03 20:50:53.261956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:53.261924 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q"] Nov 03 20:50:53.266248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:53.266225 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-ks68q"] Nov 03 20:50:53.736033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:53.736000 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:54.501110 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:54.501079 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:54.682494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:54.682467 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" path="/var/lib/kubelet/pods/7c8f34d6-5147-405a-8c9f-ea912c4b6b34/volumes" Nov 03 20:50:55.313763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:55.313715 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:56.090591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:56.090553 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:56.918549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:56.918512 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:57.665521 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:57.664169 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:59.123224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:59.123173 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:59.261667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:59.261631 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:50:59.542715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:50:59.542669 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:00.044242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:00.044203 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:00.284811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:00.284770 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:00.709710 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:00.709665 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:01.500352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:01.500312 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:02.208020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:02.207974 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:02.920768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:02.920724 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:03.723402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:03.723263 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:04.471160 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:04.470439 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:05.214620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:05.214584 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:05.518535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:05.518501 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:06.074356 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:06.074322 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:06.813114 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:06.813078 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:07.598658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:07.598623 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:08.450643 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:08.450604 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:09.206680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:09.206638 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:10.025200 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:10.025162 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:10.699886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:10.699856 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d"] Nov 03 20:51:10.753132 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:10.753086 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:10.899505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:10.899416 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:11.040717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:11.040673 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:11.366554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:11.366432 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:11.520477 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:11.519349 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:12.108177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:12.108135 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:12.790165 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:12.790123 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:13.537986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:13.530988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:14.227836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:14.227795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:15.158409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:15.158373 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:15.700564 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:15.700490 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" podUID="af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" containerName="payload-processing" containerID="cri-o://ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4" gracePeriod=25 Nov 03 20:51:15.997761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:15.997735 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:51:16.011156 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.011129 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:16.076542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.076504 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-provider-credentials\") pod \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " Nov 03 20:51:16.077365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.077304 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-kube-api-access-nbckk\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-kube-api-access-nbckk\") pod \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " Nov 03 20:51:16.077997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.077964 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-plugins-config-volume\") pod \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " Nov 03 20:51:16.080440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.078519 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-routing-overlay-volume\") pod \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\" (UID: \"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb\") " Nov 03 20:51:16.109876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.079844 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-plugins-config-volume" pod "af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" (UID: "af1bfdda-f732-4b1e-bbb0-354bbfcf22fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:16.109876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.080077 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-routing-overlay-volume" pod "af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" (UID: "af1bfdda-f732-4b1e-bbb0-354bbfcf22fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:16.121713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.121681 4711 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-routing-overlay-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:16.121950 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.121933 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:16.145489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.125948 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-provider-credentials" pod "af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" (UID: "af1bfdda-f732-4b1e-bbb0-354bbfcf22fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:16.145489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.137081 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-kube-api-access-nbckk" pod "af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" (UID: "af1bfdda-f732-4b1e-bbb0-354bbfcf22fb"). InnerVolumeSpecName "kube-api-access-nbckk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:16.255946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.245659 4711 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-provider-credentials\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:16.255946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.245720 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbckk\" (UniqueName: \"kubernetes.io/projected/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb-kube-api-access-nbckk\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:16.446175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.446104 4711 generic.go:356] "Generic (PLEG): container finished" podID="af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" containerID="ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4" exitCode=0 Nov 03 20:51:16.446175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.446149 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" Nov 03 20:51:16.446175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.446170 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" event={"ID":"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb","Type":"ContainerDied","Data":"ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4"} Nov 03 20:51:16.446403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.446210 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d" event={"ID":"af1bfdda-f732-4b1e-bbb0-354bbfcf22fb","Type":"ContainerDied","Data":"833ebd85f006c2bea21ff481d68f440b09136cf46b73c8bca03790673cd77bd1"} Nov 03 20:51:16.446403 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.446233 4711 scope.go:122] "RemoveContainer" containerID="ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4" Nov 03 20:51:16.458808 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.458785 4711 scope.go:122] "RemoveContainer" containerID="ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4" Nov 03 20:51:16.459083 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:51:16.459056 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4\": container with ID starting with ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4 not found: ID does not exist" containerID="ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4" Nov 03 20:51:16.459140 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.459094 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4"} err="failed to get container status \"ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4\": rpc error: code = NotFound desc = could not find container \"ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4\": container with ID starting with ab8cf81fb5c113689d317dc719131221806baf5e3da543583931f27c02f182a4 not found: ID does not exist" Nov 03 20:51:16.473381 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.473358 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d"] Nov 03 20:51:16.477428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.477397 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-dce009/payload-processing-external-model-e2e-extmod-a-dce009-8659bwr8d"] Nov 03 20:51:16.684193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.684101 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" path="/var/lib/kubelet/pods/af1bfdda-f732-4b1e-bbb0-354bbfcf22fb/volumes" Nov 03 20:51:16.806030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:16.805996 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:17.536056 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:51:17.536018 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:51:17.716223 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:17.716186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:18.160620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:18.160588 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m"] Nov 03 20:51:18.160861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:18.160837 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" containerID="cri-o://094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603" gracePeriod=30 Nov 03 20:51:18.227735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:18.227701 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:18.366021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:18.365961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:18.513465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:18.507624 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:18.738632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:18.738594 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:19.198908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:19.198877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:19.370467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:19.365933 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:20.117616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:20.117578 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:20.488367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:20.488246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:20.964870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:20.964826 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:21.766214 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:21.766181 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:51:21.766689 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:21.766259 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:51:22.066020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:22.065937 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:22.721400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:22.721362 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:22.879169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:22.879132 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:23.019748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.018376 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:23.446939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.446914 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:51:23.596939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.596731 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f21b89-0622-4fc8-84c5-8635f34544ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f21b89-0622-4fc8-84c5-8635f34544ba-istiod-ca-cert\") pod \"b8f21b89-0622-4fc8-84c5-8635f34544ba\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " Nov 03 20:51:23.596939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.596929 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-data\") pod \"b8f21b89-0622-4fc8-84c5-8635f34544ba\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " Nov 03 20:51:23.597217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.597124 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-envoy\") pod \"b8f21b89-0622-4fc8-84c5-8635f34544ba\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " Nov 03 20:51:23.597217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.597149 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-podinfo\") pod \"b8f21b89-0622-4fc8-84c5-8635f34544ba\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " Nov 03 20:51:23.597217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.597201 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-certs\") pod \"b8f21b89-0622-4fc8-84c5-8635f34544ba\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " Nov 03 20:51:23.597425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.597235 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-socket\") pod \"b8f21b89-0622-4fc8-84c5-8635f34544ba\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " Nov 03 20:51:23.597425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.597232 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f21b89-0622-4fc8-84c5-8635f34544ba-istiod-ca-cert" pod "b8f21b89-0622-4fc8-84c5-8635f34544ba" (UID: "b8f21b89-0622-4fc8-84c5-8635f34544ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:23.597425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.597308 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-credential-socket\") pod \"b8f21b89-0622-4fc8-84c5-8635f34544ba\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " Nov 03 20:51:23.597425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.597406 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-kube-api-access-zdsl6\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-kube-api-access-zdsl6\") pod \"b8f21b89-0622-4fc8-84c5-8635f34544ba\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " Nov 03 20:51:23.597780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.597544 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-token\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-token\") pod \"b8f21b89-0622-4fc8-84c5-8635f34544ba\" (UID: \"b8f21b89-0622-4fc8-84c5-8635f34544ba\") " Nov 03 20:51:23.597780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.597649 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-certs" pod "b8f21b89-0622-4fc8-84c5-8635f34544ba" (UID: "b8f21b89-0622-4fc8-84c5-8635f34544ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:23.598128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.598106 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-data" pod "b8f21b89-0622-4fc8-84c5-8635f34544ba" (UID: "b8f21b89-0622-4fc8-84c5-8635f34544ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:23.598193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.598158 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f21b89-0622-4fc8-84c5-8635f34544ba-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.598193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.598175 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.598296 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.598278 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-credential-socket" pod "b8f21b89-0622-4fc8-84c5-8635f34544ba" (UID: "b8f21b89-0622-4fc8-84c5-8635f34544ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:23.598486 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.598465 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-socket" pod "b8f21b89-0622-4fc8-84c5-8635f34544ba" (UID: "b8f21b89-0622-4fc8-84c5-8635f34544ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:23.599782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.599758 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-envoy" pod "b8f21b89-0622-4fc8-84c5-8635f34544ba" (UID: "b8f21b89-0622-4fc8-84c5-8635f34544ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:23.600580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.600553 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-kube-api-access-zdsl6" pod "b8f21b89-0622-4fc8-84c5-8635f34544ba" (UID: "b8f21b89-0622-4fc8-84c5-8635f34544ba"). InnerVolumeSpecName "kube-api-access-zdsl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:23.600699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.600682 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-token" pod "b8f21b89-0622-4fc8-84c5-8635f34544ba" (UID: "b8f21b89-0622-4fc8-84c5-8635f34544ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:23.600823 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.600807 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-podinfo" pod "b8f21b89-0622-4fc8-84c5-8635f34544ba" (UID: "b8f21b89-0622-4fc8-84c5-8635f34544ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:51:23.654073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.654042 4711 generic.go:356] "Generic (PLEG): container finished" podID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerID="094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603" exitCode=0 Nov 03 20:51:23.654218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.654093 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" Nov 03 20:51:23.654218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.654117 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" event={"ID":"b8f21b89-0622-4fc8-84c5-8635f34544ba","Type":"ContainerDied","Data":"094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603"} Nov 03 20:51:23.654218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.654172 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m" event={"ID":"b8f21b89-0622-4fc8-84c5-8635f34544ba","Type":"ContainerDied","Data":"11213ca392f391eb3c5b992de810e2b4d8faa66c049c9dcc216164679c29abfa"} Nov 03 20:51:23.654218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.654197 4711 scope.go:122] "RemoveContainer" containerID="094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603" Nov 03 20:51:23.665136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.665120 4711 scope.go:122] "RemoveContainer" containerID="094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603" Nov 03 20:51:23.665435 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:51:23.665409 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603\": container with ID starting with 094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603 not found: ID does not exist" containerID="094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603" Nov 03 20:51:23.665512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.665466 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603"} err="failed to get container status \"094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603\": rpc error: code = NotFound desc = could not find container \"094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603\": container with ID starting with 094c1f1834b1d8d744a520c1053a9c027e58512692233b3378ce8e6146188603 not found: ID does not exist" Nov 03 20:51:23.683136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.683112 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m"] Nov 03 20:51:23.689579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.689554 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-ssm7m"] Nov 03 20:51:23.699357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.699335 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.699357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.699358 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.699544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.699368 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.699544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.699377 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f21b89-0622-4fc8-84c5-8635f34544ba-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.699544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.699386 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.699544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.699394 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f21b89-0622-4fc8-84c5-8635f34544ba-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.699544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.699403 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdsl6\" (UniqueName: \"kubernetes.io/projected/b8f21b89-0622-4fc8-84c5-8635f34544ba-kube-api-access-zdsl6\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:23.700494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:23.700465 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:24.007746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:24.007714 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:24.247470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:24.239619 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:24.684177 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:24.684141 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" path="/var/lib/kubelet/pods/b8f21b89-0622-4fc8-84c5-8635f34544ba/volumes" Nov 03 20:51:24.768636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:24.768603 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:25.504279 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:25.504243 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:26.347226 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:26.347189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:27.004272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:27.004239 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:27.164879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:27.164839 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:27.578716 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:27.578671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:27.828360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:27.828327 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr"] Nov 03 20:51:27.828630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:27.828593 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" containerID="cri-o://6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c" gracePeriod=30 Nov 03 20:51:27.897859 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:27.897818 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:28.217071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:28.216913 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:28.654889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:28.654852 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:29.101488 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:29.101436 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:29.789743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:29.789704 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:30.401670 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:30.401633 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:30.544993 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:30.544959 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:30.668923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:30.668844 4711 patch_prober.go:29] interesting pod/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:51:30.668923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:30.668905 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:51:31.193645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:31.193611 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:31.514962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:31.514923 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:32.031738 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:32.031704 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:32.162408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:32.162365 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:32.920908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:32.920874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:33.201315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.201288 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:51:33.266692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.266655 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ac7766-28e7-47ae-9e68-29251a841002-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac7766-28e7-47ae-9e68-29251a841002-istiod-ca-cert\") pod \"44ac7766-28e7-47ae-9e68-29251a841002\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " Nov 03 20:51:33.266901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.266711 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-data\") pod \"44ac7766-28e7-47ae-9e68-29251a841002\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " Nov 03 20:51:33.266901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.266818 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-envoy\") pod \"44ac7766-28e7-47ae-9e68-29251a841002\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " Nov 03 20:51:33.266901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.266853 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-credential-socket\") pod \"44ac7766-28e7-47ae-9e68-29251a841002\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " Nov 03 20:51:33.267100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.266953 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-istio-token\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-istio-token\") pod \"44ac7766-28e7-47ae-9e68-29251a841002\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " Nov 03 20:51:33.267100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.267021 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44ac7766-28e7-47ae-9e68-29251a841002-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac7766-28e7-47ae-9e68-29251a841002-istio-podinfo\") pod \"44ac7766-28e7-47ae-9e68-29251a841002\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " Nov 03 20:51:33.267100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.266966 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ac7766-28e7-47ae-9e68-29251a841002-istiod-ca-cert" pod "44ac7766-28e7-47ae-9e68-29251a841002" (UID: "44ac7766-28e7-47ae-9e68-29251a841002"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:33.267100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.267034 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-data" pod "44ac7766-28e7-47ae-9e68-29251a841002" (UID: "44ac7766-28e7-47ae-9e68-29251a841002"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.267100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.267059 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-kube-api-access-hhzlf\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-kube-api-access-hhzlf\") pod \"44ac7766-28e7-47ae-9e68-29251a841002\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " Nov 03 20:51:33.267327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.267125 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-certs\") pod \"44ac7766-28e7-47ae-9e68-29251a841002\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " Nov 03 20:51:33.267327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.267175 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-credential-socket" pod "44ac7766-28e7-47ae-9e68-29251a841002" (UID: "44ac7766-28e7-47ae-9e68-29251a841002"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.267327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.267187 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-socket\") pod \"44ac7766-28e7-47ae-9e68-29251a841002\" (UID: \"44ac7766-28e7-47ae-9e68-29251a841002\") " Nov 03 20:51:33.267789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.267768 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-certs" pod "44ac7766-28e7-47ae-9e68-29251a841002" (UID: "44ac7766-28e7-47ae-9e68-29251a841002"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.268067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.268045 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-socket" pod "44ac7766-28e7-47ae-9e68-29251a841002" (UID: "44ac7766-28e7-47ae-9e68-29251a841002"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.268185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.268099 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac7766-28e7-47ae-9e68-29251a841002-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.268185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.268124 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.268185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.268139 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.268185 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.268153 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.269271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.269251 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-envoy" pod "44ac7766-28e7-47ae-9e68-29251a841002" (UID: "44ac7766-28e7-47ae-9e68-29251a841002"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:51:33.269691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.269668 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-istio-token" pod "44ac7766-28e7-47ae-9e68-29251a841002" (UID: "44ac7766-28e7-47ae-9e68-29251a841002"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:33.269764 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.269723 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-kube-api-access-hhzlf" pod "44ac7766-28e7-47ae-9e68-29251a841002" (UID: "44ac7766-28e7-47ae-9e68-29251a841002"). InnerVolumeSpecName "kube-api-access-hhzlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:33.270331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.270314 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44ac7766-28e7-47ae-9e68-29251a841002-istio-podinfo" pod "44ac7766-28e7-47ae-9e68-29251a841002" (UID: "44ac7766-28e7-47ae-9e68-29251a841002"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:51:33.369879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.369849 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.369879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.369874 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.369879 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.369883 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac7766-28e7-47ae-9e68-29251a841002-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.370101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.369894 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzlf\" (UniqueName: \"kubernetes.io/projected/44ac7766-28e7-47ae-9e68-29251a841002-kube-api-access-hhzlf\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.370101 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.369903 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac7766-28e7-47ae-9e68-29251a841002-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:33.688551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.688476 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:33.773218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.773189 4711 generic.go:356] "Generic (PLEG): container finished" podID="44ac7766-28e7-47ae-9e68-29251a841002" containerID="6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c" exitCode=0 Nov 03 20:51:33.773416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.773240 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" Nov 03 20:51:33.773416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.773273 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" event={"ID":"44ac7766-28e7-47ae-9e68-29251a841002","Type":"ContainerDied","Data":"6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c"} Nov 03 20:51:33.773416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.773339 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr" event={"ID":"44ac7766-28e7-47ae-9e68-29251a841002","Type":"ContainerDied","Data":"c48f30ee9259f421b926f5c44716a40e61e0a519a7d08f72ccf2c3259b79beac"} Nov 03 20:51:33.773416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.773369 4711 scope.go:122] "RemoveContainer" containerID="6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c" Nov 03 20:51:33.786138 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.786115 4711 scope.go:122] "RemoveContainer" containerID="6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c" Nov 03 20:51:33.786406 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:51:33.786387 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c\": container with ID starting with 6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c not found: ID does not exist" containerID="6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c" Nov 03 20:51:33.786544 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.786413 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c"} err="failed to get container status \"6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c\": rpc error: code = NotFound desc = could not find container \"6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c\": container with ID starting with 6c0f401827c14a3be12d4a090b57055f010c5369c0d182ce58ef6b1e7895685c not found: ID does not exist" Nov 03 20:51:33.800377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.800351 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr"] Nov 03 20:51:33.806455 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:33.806424 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3d4fe2-openshift-default-656d7c868c-k5ljr"] Nov 03 20:51:34.311063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:34.311026 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:34.586811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:34.586729 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn"] Nov 03 20:51:34.682145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:34.682109 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:34.687659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:34.687632 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ac7766-28e7-47ae-9e68-29251a841002" path="/var/lib/kubelet/pods/44ac7766-28e7-47ae-9e68-29251a841002/volumes" Nov 03 20:51:34.815943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:34.815909 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:35.309593 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:35.309555 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:35.445775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:35.445740 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:36.201243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:36.201215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:36.806976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:36.806942 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q"] Nov 03 20:51:36.807614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:36.807170 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" podUID="898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" containerName="maas-api" containerID="cri-o://58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62" gracePeriod=30 Nov 03 20:51:36.890915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:36.890878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:37.228531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.227831 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:37.501597 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.501513 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:51:37.506126 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.506104 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:51:37.510648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.510627 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:51:37.514598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.514577 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:51:37.517749 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.517734 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:51:37.547573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.547539 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-openshift-service-ca\") pod \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " Nov 03 20:51:37.547703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.547593 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-metrics-certs\") pod \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " Nov 03 20:51:37.547742 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.547727 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-kube-api-access-qd72p\" (UniqueName: \"kubernetes.io/projected/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-kube-api-access-qd72p\") pod \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " Nov 03 20:51:37.547854 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.547839 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-maas-api-tls\") pod \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\" (UID: \"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4\") " Nov 03 20:51:37.547989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.547964 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-openshift-service-ca" pod "898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" (UID: "898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:37.548756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.548724 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:37.549926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.549901 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-kube-api-access-qd72p" pod "898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" (UID: "898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4"). InnerVolumeSpecName "kube-api-access-qd72p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:37.550059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.550026 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-maas-api-tls" pod "898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" (UID: "898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:37.550137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.550115 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-metrics-certs" pod "898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" (UID: "898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:37.653022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.652984 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd72p\" (UniqueName: \"kubernetes.io/projected/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-kube-api-access-qd72p\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:37.653022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.653025 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:37.653244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.653038 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:37.811086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.811053 4711 generic.go:356] "Generic (PLEG): container finished" podID="898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" containerID="58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62" exitCode=0 Nov 03 20:51:37.811545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.811103 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" Nov 03 20:51:37.811545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.811122 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" event={"ID":"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4","Type":"ContainerDied","Data":"58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62"} Nov 03 20:51:37.811545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.811171 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q" event={"ID":"898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4","Type":"ContainerDied","Data":"d8dae52b8aff6785113b7b8a06ed232152a0e3d78b9128453f12c8dd286a9551"} Nov 03 20:51:37.811545 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.811198 4711 scope.go:122] "RemoveContainer" containerID="58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62" Nov 03 20:51:37.822333 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.822315 4711 scope.go:122] "RemoveContainer" containerID="58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62" Nov 03 20:51:37.822646 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:51:37.822627 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62\": container with ID starting with 58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62 not found: ID does not exist" containerID="58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62" Nov 03 20:51:37.822706 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.822653 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62"} err="failed to get container status \"58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62\": rpc error: code = NotFound desc = could not find container \"58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62\": container with ID starting with 58c3040793b579a093a776963bb127673641d96c1f6f98bdc9be84ab10b89b62 not found: ID does not exist" Nov 03 20:51:37.858966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.858937 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q"] Nov 03 20:51:37.862758 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:37.862731 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c94fdf-5fdc4df7b7-2nx8q"] Nov 03 20:51:38.090865 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:38.090783 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:38.296467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:38.287984 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq"] Nov 03 20:51:38.362536 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:38.361440 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:38.539668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:38.535592 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:38.768712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:38.768675 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" path="/var/lib/kubelet/pods/898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4/volumes" Nov 03 20:51:39.261154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:39.261121 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:39.394950 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:39.394912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:39.578877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:39.578792 4711 scope.go:122] "RemoveContainer" containerID="2160c3c559c1b62b57b8a5b118ca39b4575d3308f958447257be04a1bf385306" Nov 03 20:51:39.587513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:39.587439 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" podUID="f95e60b8-b0ce-400e-bdd7-cd9c4265943a" containerName="payload-pre-processing" containerID="cri-o://d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b" gracePeriod=25 Nov 03 20:51:39.987699 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:39.987602 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:40.729748 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:40.729712 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:41.323946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:41.323912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:41.455524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:41.455480 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:41.610522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:41.605063 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:41.792834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:41.792795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:42.119863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:42.119828 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:42.514248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:42.514218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:43.003303 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:43.003207 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:43.146087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:43.146053 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:43.315468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:43.297963 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" podUID="4a223b44-25c7-49bf-96d2-e166e1d59604" containerName="payload-processing" containerID="cri-o://d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5" gracePeriod=25 Nov 03 20:51:43.364939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:43.364904 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:44.096727 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:44.096685 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:44.798377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:44.798327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:45.328549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:45.328503 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:45.461142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:45.461107 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:45.611963 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:45.611874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:45.793681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:45.793643 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:46.168245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:46.168200 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:46.613681 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:46.613643 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:46.974836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:46.974729 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:47.103509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:47.103435 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" podUID="f95e60b8-b0ce-400e-bdd7-cd9c4265943a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:51:47.304801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:47.304768 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:47.444412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:47.444379 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:47.538772 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:51:47.538735 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:51:47.579277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:47.579185 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:48.318680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:48.318645 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:49.204926 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:49.204892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:49.631155 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:49.631112 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" podUID="4a223b44-25c7-49bf-96d2-e166e1d59604" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:51:49.891420 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:49.891333 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:50.686484 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:50.686420 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:51.414840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:51.414795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:52.225609 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:52.225575 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:52.508300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:52.508258 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:52.637603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:52.637569 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:53.099030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:53.098998 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:53.236088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:53.236052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:53.980125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:53.980071 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:54.695306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:54.695267 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:55.398624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:55.398588 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:56.202923 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:56.202887 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:56.847876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:56.847798 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:57.102106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:57.102019 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" podUID="f95e60b8-b0ce-400e-bdd7-cd9c4265943a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:51:57.538050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:57.538012 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:58.217979 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:58.217939 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:59.002469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:59.002180 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:59.217065 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:59.217029 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:59.406607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:59.406569 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:59.630912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:59.630879 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" podUID="4a223b44-25c7-49bf-96d2-e166e1d59604" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:51:59.753605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:59.753052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:51:59.975676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:51:59.975602 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:52:00.067508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.067437 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-plugins-config-volume\") pod \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\" (UID: \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\") " Nov 03 20:52:00.067957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.067737 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-plugins-config-volume" pod "f95e60b8-b0ce-400e-bdd7-cd9c4265943a" (UID: "f95e60b8-b0ce-400e-bdd7-cd9c4265943a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:00.067957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.067858 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-kube-api-access-mxqtb\" (UniqueName: \"kubernetes.io/projected/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-kube-api-access-mxqtb\") pod \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\" (UID: \"f95e60b8-b0ce-400e-bdd7-cd9c4265943a\") " Nov 03 20:52:00.068543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.068490 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:00.069815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.069794 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-kube-api-access-mxqtb" pod "f95e60b8-b0ce-400e-bdd7-cd9c4265943a" (UID: "f95e60b8-b0ce-400e-bdd7-cd9c4265943a"). InnerVolumeSpecName "kube-api-access-mxqtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:00.094150 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.094121 4711 generic.go:356] "Generic (PLEG): container finished" podID="f95e60b8-b0ce-400e-bdd7-cd9c4265943a" containerID="d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b" exitCode=0 Nov 03 20:52:00.094287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.094166 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" Nov 03 20:52:00.094287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.094209 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" event={"ID":"f95e60b8-b0ce-400e-bdd7-cd9c4265943a","Type":"ContainerDied","Data":"d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b"} Nov 03 20:52:00.094374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.094298 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn" event={"ID":"f95e60b8-b0ce-400e-bdd7-cd9c4265943a","Type":"ContainerDied","Data":"ad050739bb2231b0cbc2a252b271652684bcbdf78db7c91f7e24c0a2bb2e3a1e"} Nov 03 20:52:00.094374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.094329 4711 scope.go:122] "RemoveContainer" containerID="d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b" Nov 03 20:52:00.105912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.105896 4711 scope.go:122] "RemoveContainer" containerID="d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b" Nov 03 20:52:00.106247 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:52:00.106188 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b\": container with ID starting with d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b not found: ID does not exist" containerID="d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b" Nov 03 20:52:00.106299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.106262 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b"} err="failed to get container status \"d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b\": rpc error: code = NotFound desc = could not find container \"d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b\": container with ID starting with d509baf72ff922f021ace042180fbd3e2fc387bb14fb788374c167ff119f129b not found: ID does not exist" Nov 03 20:52:00.121982 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.121942 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn"] Nov 03 20:52:00.127353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.127326 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c94fdf-6877495847-s5mkn"] Nov 03 20:52:00.170587 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.170554 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxqtb\" (UniqueName: \"kubernetes.io/projected/f95e60b8-b0ce-400e-bdd7-cd9c4265943a-kube-api-access-mxqtb\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:00.333989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.333958 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:00.579983 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.579950 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:00.682178 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:00.682104 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95e60b8-b0ce-400e-bdd7-cd9c4265943a" path="/var/lib/kubelet/pods/f95e60b8-b0ce-400e-bdd7-cd9c4265943a/volumes" Nov 03 20:52:01.239343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:01.239307 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:01.921119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:01.921078 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:02.705635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:02.697521 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:03.428861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:03.428827 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:03.676071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:03.676047 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:52:03.736151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:03.736120 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a223b44-25c7-49bf-96d2-e166e1d59604-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a223b44-25c7-49bf-96d2-e166e1d59604-plugins-config-volume\") pod \"4a223b44-25c7-49bf-96d2-e166e1d59604\" (UID: \"4a223b44-25c7-49bf-96d2-e166e1d59604\") " Nov 03 20:52:03.736636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:03.736228 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a223b44-25c7-49bf-96d2-e166e1d59604-kube-api-access-85b72\" (UniqueName: \"kubernetes.io/projected/4a223b44-25c7-49bf-96d2-e166e1d59604-kube-api-access-85b72\") pod \"4a223b44-25c7-49bf-96d2-e166e1d59604\" (UID: \"4a223b44-25c7-49bf-96d2-e166e1d59604\") " Nov 03 20:52:03.736636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:03.736564 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a223b44-25c7-49bf-96d2-e166e1d59604-plugins-config-volume" pod "4a223b44-25c7-49bf-96d2-e166e1d59604" (UID: "4a223b44-25c7-49bf-96d2-e166e1d59604"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:03.737235 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:03.737215 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a223b44-25c7-49bf-96d2-e166e1d59604-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:03.738547 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:03.738524 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a223b44-25c7-49bf-96d2-e166e1d59604-kube-api-access-85b72" pod "4a223b44-25c7-49bf-96d2-e166e1d59604" (UID: "4a223b44-25c7-49bf-96d2-e166e1d59604"). InnerVolumeSpecName "kube-api-access-85b72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:03.840483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:03.839657 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85b72\" (UniqueName: \"kubernetes.io/projected/4a223b44-25c7-49bf-96d2-e166e1d59604-kube-api-access-85b72\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:04.127756 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.127719 4711 generic.go:356] "Generic (PLEG): container finished" podID="4a223b44-25c7-49bf-96d2-e166e1d59604" containerID="d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5" exitCode=0 Nov 03 20:52:04.127976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.127779 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" Nov 03 20:52:04.127976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.127801 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" event={"ID":"4a223b44-25c7-49bf-96d2-e166e1d59604","Type":"ContainerDied","Data":"d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5"} Nov 03 20:52:04.127976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.127887 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq" event={"ID":"4a223b44-25c7-49bf-96d2-e166e1d59604","Type":"ContainerDied","Data":"3d4326047fdd551c25c6158727e5f66be66c303983b68ac5a43ebc1ff04756d0"} Nov 03 20:52:04.127976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.127915 4711 scope.go:122] "RemoveContainer" containerID="d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5" Nov 03 20:52:04.138973 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.138957 4711 scope.go:122] "RemoveContainer" containerID="d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5" Nov 03 20:52:04.139206 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:52:04.139189 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5\": container with ID starting with d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5 not found: ID does not exist" containerID="d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5" Nov 03 20:52:04.139248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.139213 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5"} err="failed to get container status \"d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5\": rpc error: code = NotFound desc = could not find container \"d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5\": container with ID starting with d2a42d7a79fe0e2e835c8663f7b0bb98698bcd1af6739b167e3d58002b53dcb5 not found: ID does not exist" Nov 03 20:52:04.153158 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.153134 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq"] Nov 03 20:52:04.159277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.159253 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c6934-7d5b684686-lcrhq"] Nov 03 20:52:04.199330 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.199304 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:04.683078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.683045 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a223b44-25c7-49bf-96d2-e166e1d59604" path="/var/lib/kubelet/pods/4a223b44-25c7-49bf-96d2-e166e1d59604/volumes" Nov 03 20:52:04.894430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:04.894386 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:05.613883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:05.613848 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:06.267206 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:06.267171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:07.022560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:07.022517 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:08.208556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:08.208528 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:09.272217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:09.272182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:10.193466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:10.193414 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:11.456880 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:11.456846 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:12.598112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:12.598075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:13.416073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:13.416039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:13.679555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:13.679471 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:14.107637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:14.107595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:14.237457 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:14.237405 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:14.592834 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:14.592798 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:14.822271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:14.822162 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:15.198145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:15.198109 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:15.912639 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:15.912600 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:16.518281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:16.518247 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:16.755471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:16.755422 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:17.536307 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:52:17.536270 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:52:17.748788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:17.748734 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:18.422798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:18.422752 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:18.608844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:18.608811 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:18.767054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:18.767022 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:19.263518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:19.255941 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:19.815107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:19.815073 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:19.957139 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:19.957103 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:20.106522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:20.104028 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:20.915753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:20.915715 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:21.595203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:21.595172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:22.324898 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:22.324854 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:23.008910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:23.008873 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:23.708078 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:23.708032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:24.319324 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:24.319275 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:24.463918 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:24.463880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:24.812352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:24.812305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:25.110406 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:25.110318 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n"] Nov 03 20:52:25.110632 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:25.110602 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" containerID="cri-o://fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2" gracePeriod=30 Nov 03 20:52:25.184621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:25.184584 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:25.319242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:25.319199 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:26.164331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:26.164292 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:26.297991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:26.297953 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:26.448525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:26.442986 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:26.638159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:26.638124 4711 patch_prober.go:29] interesting pod/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:52:26.638346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:26.638196 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:52:27.007302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:27.007266 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:27.273072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:27.272980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:28.111586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:28.111541 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:28.796095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:28.796055 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:29.491286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:29.482777 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:30.209434 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.209395 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:30.472827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.472803 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:52:30.514162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514134 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-envoy\") pod \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " Nov 03 20:52:30.514335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514192 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-kube-api-access-5dbwf\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-kube-api-access-5dbwf\") pod \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " Nov 03 20:52:30.514335 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514251 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-data\") pod \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " Nov 03 20:52:30.514493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514345 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-certs\") pod \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " Nov 03 20:52:30.514493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514429 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-credential-socket\") pod \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " Nov 03 20:52:30.514493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514486 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-token\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-token\") pod \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " Nov 03 20:52:30.514671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514525 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-socket\") pod \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " Nov 03 20:52:30.514671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514572 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-podinfo\") pod \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " Nov 03 20:52:30.514778 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514695 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istiod-ca-cert\") pod \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\" (UID: \"84ed0c60-5fec-4a41-912b-f5f2e9ff1516\") " Nov 03 20:52:30.514835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514784 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-credential-socket" pod "84ed0c60-5fec-4a41-912b-f5f2e9ff1516" (UID: "84ed0c60-5fec-4a41-912b-f5f2e9ff1516"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:30.515812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.514974 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-certs" pod "84ed0c60-5fec-4a41-912b-f5f2e9ff1516" (UID: "84ed0c60-5fec-4a41-912b-f5f2e9ff1516"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:30.515812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.515301 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-data" pod "84ed0c60-5fec-4a41-912b-f5f2e9ff1516" (UID: "84ed0c60-5fec-4a41-912b-f5f2e9ff1516"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:30.515812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.515304 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istiod-ca-cert" pod "84ed0c60-5fec-4a41-912b-f5f2e9ff1516" (UID: "84ed0c60-5fec-4a41-912b-f5f2e9ff1516"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:30.515812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.515756 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.515812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.515778 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.515812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.515777 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-socket" pod "84ed0c60-5fec-4a41-912b-f5f2e9ff1516" (UID: "84ed0c60-5fec-4a41-912b-f5f2e9ff1516"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:30.515812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.515792 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.515812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.515807 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.517087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.517062 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-kube-api-access-5dbwf" pod "84ed0c60-5fec-4a41-912b-f5f2e9ff1516" (UID: "84ed0c60-5fec-4a41-912b-f5f2e9ff1516"). InnerVolumeSpecName "kube-api-access-5dbwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:30.518592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.518561 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-envoy" pod "84ed0c60-5fec-4a41-912b-f5f2e9ff1516" (UID: "84ed0c60-5fec-4a41-912b-f5f2e9ff1516"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:30.519315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.519288 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-podinfo" pod "84ed0c60-5fec-4a41-912b-f5f2e9ff1516" (UID: "84ed0c60-5fec-4a41-912b-f5f2e9ff1516"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:52:30.519844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.519819 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-token" pod "84ed0c60-5fec-4a41-912b-f5f2e9ff1516" (UID: "84ed0c60-5fec-4a41-912b-f5f2e9ff1516"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:30.565499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.565465 4711 generic.go:356] "Generic (PLEG): container finished" podID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerID="fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2" exitCode=0 Nov 03 20:52:30.565659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.565543 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" Nov 03 20:52:30.565659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.565562 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" event={"ID":"84ed0c60-5fec-4a41-912b-f5f2e9ff1516","Type":"ContainerDied","Data":"fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2"} Nov 03 20:52:30.565780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.565658 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n" event={"ID":"84ed0c60-5fec-4a41-912b-f5f2e9ff1516","Type":"ContainerDied","Data":"34a8a4ba475d71fa963542a0e3ff1ca3f451ee671237b0b21eed9e26d34c93bf"} Nov 03 20:52:30.565780 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.565693 4711 scope.go:122] "RemoveContainer" containerID="fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2" Nov 03 20:52:30.576871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.576849 4711 scope.go:122] "RemoveContainer" containerID="560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115" Nov 03 20:52:30.587773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.587753 4711 scope.go:122] "RemoveContainer" containerID="fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2" Nov 03 20:52:30.588045 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:52:30.588026 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2\": container with ID starting with fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2 not found: ID does not exist" containerID="fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2" Nov 03 20:52:30.588153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.588055 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2"} err="failed to get container status \"fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2\": rpc error: code = NotFound desc = could not find container \"fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2\": container with ID starting with fc26b4d691244fc499a9a5837ef1fb88eff9293d55a3ebdf693695b0ea1c5af2 not found: ID does not exist" Nov 03 20:52:30.588153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.588072 4711 scope.go:122] "RemoveContainer" containerID="560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115" Nov 03 20:52:30.588318 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:52:30.588298 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115\": container with ID starting with 560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115 not found: ID does not exist" containerID="560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115" Nov 03 20:52:30.588416 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.588322 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115"} err="failed to get container status \"560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115\": rpc error: code = NotFound desc = could not find container \"560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115\": container with ID starting with 560f376a761c3ef37ddb31b73a8a63859ef7f770f829776d3c90e2c291cad115 not found: ID does not exist" Nov 03 20:52:30.592046 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.592026 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n"] Nov 03 20:52:30.600730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.600709 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f5afd7-openshift-default-f54dc7459-sct9n"] Nov 03 20:52:30.616876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.616856 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.616960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.616884 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dbwf\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-kube-api-access-5dbwf\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.616960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.616895 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.616960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.616904 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.616960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.616912 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84ed0c60-5fec-4a41-912b-f5f2e9ff1516-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.682163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.682139 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" path="/var/lib/kubelet/pods/84ed0c60-5fec-4a41-912b-f5f2e9ff1516/volumes" Nov 03 20:52:30.891713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:30.891681 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:31.623353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:31.623317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:32.297594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:32.297562 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:33.089298 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:33.089218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:33.817935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:33.817905 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:34.594276 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:34.594236 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:35.372029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:35.371991 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:35.996636 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:35.996602 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:36.692476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:36.692425 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:37.543280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:37.539332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:38.289313 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:38.289278 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:38.921219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:38.921179 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:39.513989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.513953 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt"] Nov 03 20:52:39.515170 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515147 4711 cpu_manager.go:414] "containerMap: removing container" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" Nov 03 20:52:39.515256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515174 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" Nov 03 20:52:39.515256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515198 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" Nov 03 20:52:39.515256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515206 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" Nov 03 20:52:39.515256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515219 4711 cpu_manager.go:414] "containerMap: removing container" podUID="af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" containerName="payload-processing" Nov 03 20:52:39.515256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515227 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" containerName="payload-processing" Nov 03 20:52:39.515256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515255 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f95e60b8-b0ce-400e-bdd7-cd9c4265943a" containerName="payload-pre-processing" Nov 03 20:52:39.515485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515263 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95e60b8-b0ce-400e-bdd7-cd9c4265943a" containerName="payload-pre-processing" Nov 03 20:52:39.515485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515285 4711 cpu_manager.go:414] "containerMap: removing container" podUID="898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" containerName="maas-api" Nov 03 20:52:39.515485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515292 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" containerName="maas-api" Nov 03 20:52:39.515485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515314 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" Nov 03 20:52:39.515485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515321 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" Nov 03 20:52:39.515485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515349 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4a223b44-25c7-49bf-96d2-e166e1d59604" containerName="payload-processing" Nov 03 20:52:39.515485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515357 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a223b44-25c7-49bf-96d2-e166e1d59604" containerName="payload-processing" Nov 03 20:52:39.515485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515384 4711 cpu_manager.go:414] "containerMap: removing container" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" Nov 03 20:52:39.515485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515393 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" Nov 03 20:52:39.515785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515595 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" Nov 03 20:52:39.515785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515613 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c8f34d6-5147-405a-8c9f-ea912c4b6b34" containerName="istio-proxy" Nov 03 20:52:39.515785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515623 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1bfdda-f732-4b1e-bbb0-354bbfcf22fb" containerName="payload-processing" Nov 03 20:52:39.515785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515635 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95e60b8-b0ce-400e-bdd7-cd9c4265943a" containerName="payload-pre-processing" Nov 03 20:52:39.515785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515646 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="898bc0cd-38ea-46d6-ac51-a1c4cca1ebc4" containerName="maas-api" Nov 03 20:52:39.515785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515659 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f21b89-0622-4fc8-84c5-8635f34544ba" containerName="istio-proxy" Nov 03 20:52:39.515785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515673 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a223b44-25c7-49bf-96d2-e166e1d59604" containerName="payload-processing" Nov 03 20:52:39.515785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.515688 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ac7766-28e7-47ae-9e68-29251a841002" containerName="istio-proxy" Nov 03 20:52:39.520703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.520679 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.526157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.526133 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5ecee8-serving-cert\"" Nov 03 20:52:39.529974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.529951 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt"] Nov 03 20:52:39.586126 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.586084 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:39.661263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.660474 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-metrics-certs\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.661263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.660541 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/437f7610-bc66-4f26-9a3f-b751fb523aaf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.661263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.660646 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmjb\" (UniqueName: \"kubernetes.io/projected/437f7610-bc66-4f26-9a3f-b751fb523aaf-kube-api-access-6tmjb\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.661263 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.660734 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.723973 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.723898 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:39.762118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.762089 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-metrics-certs\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.762229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.762140 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/437f7610-bc66-4f26-9a3f-b751fb523aaf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.762280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.762227 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmjb\" (UniqueName: \"kubernetes.io/projected/437f7610-bc66-4f26-9a3f-b751fb523aaf-kube-api-access-6tmjb\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.762334 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.762315 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.762962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.762941 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/437f7610-bc66-4f26-9a3f-b751fb523aaf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.764670 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.764621 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-maas-api-tls\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.764760 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.764670 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-metrics-certs\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.774816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.774795 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmjb\" (UniqueName: \"kubernetes.io/projected/437f7610-bc66-4f26-9a3f-b751fb523aaf-kube-api-access-6tmjb\") pod \"maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.844558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.844530 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:39.862141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:39.862109 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:40.152522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:40.152095 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt"] Nov 03 20:52:40.254493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:40.252791 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:52:40.507706 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:40.507670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:40.795069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:40.794988 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" event={"ID":"437f7610-bc66-4f26-9a3f-b751fb523aaf","Type":"ContainerStarted","Data":"518996b3a26da6b5d16edffbb71357b7ceb3cb753594f682013f65a7d64bb8bc"} Nov 03 20:52:41.418934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:41.418896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:41.803889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:41.803857 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" event={"ID":"437f7610-bc66-4f26-9a3f-b751fb523aaf","Type":"ContainerStarted","Data":"f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1"} Nov 03 20:52:41.804096 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:41.804079 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:41.825415 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:41.825369 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" podStartSLOduration=2.272101256 podStartE2EDuration="2.825356666s" totalImagesPullingTime="553.25541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:41.823870337 +0000 UTC m=+3065.273976396" watchObservedRunningTime="2026-11-03 20:52:41.825356666 +0000 UTC m=+3065.275462715" Nov 03 20:52:42.144242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:42.144163 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:42.812144 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:42.812106 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:43.590438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:43.590404 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:44.256870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:44.256830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:44.932870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:44.932829 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:45.619013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:45.618970 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:46.296259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:46.296224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:46.961558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:46.961524 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:47.532624 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:52:47.532580 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:52:47.818020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:47.817956 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:52:47.821045 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:47.821019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:48.494386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:48.494350 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:49.194529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:49.194478 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:49.914326 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:49.914290 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:50.650812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:50.650777 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:51.319568 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:51.319516 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:52.205543 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:52.205503 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:52.861409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:52.861368 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:53.025120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:53.025082 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:53.603424 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:53.603385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:54.405888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:54.405841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:55.101910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:55.101871 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:55.916387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:55.916332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:56.622902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:56.622855 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:57.322555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:57.322514 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:58.084309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:58.084272 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:58.733625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:58.733584 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:59.269090 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:59.269048 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:59.501742 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:59.501689 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:52:59.949184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:52:59.949149 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:00.254363 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:00.254328 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:00.422468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:00.419947 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:01.160471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:01.157503 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:02.049067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:02.049032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:03.193668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:03.193628 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:03.916868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:03.916837 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:04.435306 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:04.435267 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:04.787842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:04.787795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:05.585238 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:05.585185 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:05.771985 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:05.771951 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:06.514868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:06.514835 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:07.366002 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:07.365962 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:08.122119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:08.122085 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:08.861736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:08.861689 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:09.547641 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:09.547597 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:10.318886 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:10.318822 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:11.043378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:11.043344 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:11.736717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:11.736677 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:12.523234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:12.509008 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:13.212819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:13.212658 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:13.920003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:13.919963 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:14.700881 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:14.700823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:15.394224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:15.394185 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:15.802746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:15.802711 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:16.302194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:16.302138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:16.994655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:16.994604 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:17.290992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:17.290946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:17.534379 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:53:17.534343 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:53:17.725837 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:17.725733 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:18.521034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:18.520989 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:19.217149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:19.217114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:19.217422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:19.217377 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm"] Nov 03 20:53:19.217664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:19.217640 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" containerID="cri-o://65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf" gracePeriod=30 Nov 03 20:53:20.024607 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:20.024574 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:20.794438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:20.794398 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:21.687257 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:21.687201 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:22.423328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:22.423294 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:23.206048 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:23.206008 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:23.673380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:23.673335 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:23.934167 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:23.934074 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:24.501840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.501770 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:53:24.655209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655176 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-kube-api-access-hzt55\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-kube-api-access-hzt55\") pod \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " Nov 03 20:53:24.655411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655233 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-certs\") pod \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " Nov 03 20:53:24.655411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655307 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-credential-socket\") pod \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " Nov 03 20:53:24.655411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655387 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-data\") pod \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " Nov 03 20:53:24.655666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655493 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istiod-ca-cert\") pod \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " Nov 03 20:53:24.655666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655545 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-certs" pod "a0b58e1c-d802-4a1d-9833-c2557d8128e4" (UID: "a0b58e1c-d802-4a1d-9833-c2557d8128e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:24.655666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655555 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-podinfo\") pod \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " Nov 03 20:53:24.655666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655615 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-envoy\") pod \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " Nov 03 20:53:24.655876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655689 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-data" pod "a0b58e1c-d802-4a1d-9833-c2557d8128e4" (UID: "a0b58e1c-d802-4a1d-9833-c2557d8128e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:24.655876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655722 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-credential-socket" pod "a0b58e1c-d802-4a1d-9833-c2557d8128e4" (UID: "a0b58e1c-d802-4a1d-9833-c2557d8128e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:24.655876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655732 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-token\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-token\") pod \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " Nov 03 20:53:24.656060 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.655883 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istiod-ca-cert" pod "a0b58e1c-d802-4a1d-9833-c2557d8128e4" (UID: "a0b58e1c-d802-4a1d-9833-c2557d8128e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:24.656060 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.656024 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-socket\") pod \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\" (UID: \"a0b58e1c-d802-4a1d-9833-c2557d8128e4\") " Nov 03 20:53:24.656372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.656347 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-socket" pod "a0b58e1c-d802-4a1d-9833-c2557d8128e4" (UID: "a0b58e1c-d802-4a1d-9833-c2557d8128e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:24.656788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.656765 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:24.656882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.656791 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:24.656882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.656807 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:24.656882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.656821 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:24.656882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.656835 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:24.657725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.657698 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-envoy" pod "a0b58e1c-d802-4a1d-9833-c2557d8128e4" (UID: "a0b58e1c-d802-4a1d-9833-c2557d8128e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:24.657806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.657767 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-token" pod "a0b58e1c-d802-4a1d-9833-c2557d8128e4" (UID: "a0b58e1c-d802-4a1d-9833-c2557d8128e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:24.657976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.657959 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-kube-api-access-hzt55" pod "a0b58e1c-d802-4a1d-9833-c2557d8128e4" (UID: "a0b58e1c-d802-4a1d-9833-c2557d8128e4"). InnerVolumeSpecName "kube-api-access-hzt55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:24.657976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.657965 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-podinfo" pod "a0b58e1c-d802-4a1d-9833-c2557d8128e4" (UID: "a0b58e1c-d802-4a1d-9833-c2557d8128e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:53:24.758427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.758386 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzt55\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-kube-api-access-hzt55\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:24.758427 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.758426 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:24.758642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.758457 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:24.758642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.758472 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b58e1c-d802-4a1d-9833-c2557d8128e4-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:24.800654 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:24.800616 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:25.277744 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.277713 4711 generic.go:356] "Generic (PLEG): container finished" podID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerID="65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf" exitCode=0 Nov 03 20:53:25.277942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.277760 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" Nov 03 20:53:25.277942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.277797 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" event={"ID":"a0b58e1c-d802-4a1d-9833-c2557d8128e4","Type":"ContainerDied","Data":"65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf"} Nov 03 20:53:25.277942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.277851 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm" event={"ID":"a0b58e1c-d802-4a1d-9833-c2557d8128e4","Type":"ContainerDied","Data":"1af5dfb7d9c948fc88bfba6228a82bf78ea73a61f378ec3c1d6d1ef74cd36d52"} Nov 03 20:53:25.277942 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.277881 4711 scope.go:122] "RemoveContainer" containerID="65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf" Nov 03 20:53:25.288662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.288642 4711 scope.go:122] "RemoveContainer" containerID="65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf" Nov 03 20:53:25.288958 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:53:25.288939 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf\": container with ID starting with 65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf not found: ID does not exist" containerID="65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf" Nov 03 20:53:25.289011 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.288971 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf"} err="failed to get container status \"65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf\": rpc error: code = NotFound desc = could not find container \"65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf\": container with ID starting with 65fbd572ccfad2decb135a8f66bfd500df67f60264177bf207353599657b80cf not found: ID does not exist" Nov 03 20:53:25.300798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.300769 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm"] Nov 03 20:53:25.306419 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.306392 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-rr5jm"] Nov 03 20:53:25.494394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:25.494357 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:26.206888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:26.206845 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:26.682730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:26.682695 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" path="/var/lib/kubelet/pods/a0b58e1c-d802-4a1d-9833-c2557d8128e4/volumes" Nov 03 20:53:26.936244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:26.936142 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:27.801715 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:27.801679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:28.499943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:28.499892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:29.315813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:29.308203 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:30.102121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:30.102084 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:30.764409 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:30.764370 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:31.507833 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:31.507803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:32.206504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:32.206468 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:32.923494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:32.923433 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:33.604366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:33.604324 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:34.270884 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:34.270848 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:35.011203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:35.011164 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:35.669169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:35.669134 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:36.322854 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:36.322808 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:36.590485 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:36.590383 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:37.094767 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:37.094731 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:37.874625 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:37.865665 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:38.493417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:38.493376 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:39.197740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:39.197695 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:39.914329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:39.914290 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:40.767026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:40.766985 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:41.453171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:41.453107 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:42.238577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:42.238539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:42.990053 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:42.990015 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:43.697797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:43.697757 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:44.486297 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:44.486254 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:45.194774 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:45.194737 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:45.981474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:45.978961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:46.965180 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:46.965142 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:47.750470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:47.747041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:47.933472 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:53:47.916646 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:53:48.187921 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:48.187885 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:48.324121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:48.324081 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:49.314470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:49.313521 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:50.399766 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:50.399725 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:51.298365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:51.298328 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:52.093848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:52.093804 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:52.945467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:52.944591 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:53.814492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:53.813039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:53.947219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:53.947169 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:54.745016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:54.744976 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:54.911100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:54.911063 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:55.525198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:55.525164 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:56.295724 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:56.295668 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:56.652755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:56.652659 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:57.142571 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:57.142508 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:58.025081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:58.025042 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:58.632721 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:58.632684 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:59.323822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:59.323786 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:53:59.466225 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:53:59.466189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:00.154107 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:00.154065 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:00.447615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:00.447533 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:00.817352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:00.817319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:00.953068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:00.953030 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:01.596662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:01.596628 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:02.257698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:02.257659 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:03.003670 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:03.003633 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:03.806034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:03.806002 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:04.549480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:04.547649 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:05.250475 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:05.249760 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:06.201901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:06.201871 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:06.888295 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:06.888251 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:07.681195 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:07.681166 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:08.420977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:08.420946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:09.057831 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:09.057799 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:09.547779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:09.547747 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5"] Nov 03 20:54:09.548288 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:09.548234 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" podUID="4aa7c6ac-d377-450b-820b-19fb08c44a55" containerName="maas-api" containerID="cri-o://8402d2fce6d542a58342da8ee62c4f87e6f6adc6e49a4cb7571028499dd3cad4" gracePeriod=30 Nov 03 20:54:09.630431 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:09.630400 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:10.105806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.105765 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:10.815341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.815316 4711 generic.go:356] "Generic (PLEG): container finished" podID="4aa7c6ac-d377-450b-820b-19fb08c44a55" containerID="8402d2fce6d542a58342da8ee62c4f87e6f6adc6e49a4cb7571028499dd3cad4" exitCode=0 Nov 03 20:54:10.815528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.815404 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" event={"ID":"4aa7c6ac-d377-450b-820b-19fb08c44a55","Type":"ContainerDied","Data":"8402d2fce6d542a58342da8ee62c4f87e6f6adc6e49a4cb7571028499dd3cad4"} Nov 03 20:54:10.815528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.815487 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" event={"ID":"4aa7c6ac-d377-450b-820b-19fb08c44a55","Type":"ContainerDied","Data":"d9471bdaa4b35db888077c892efa437eff5e6f6f4afdab1d030bf1836cf7d299"} Nov 03 20:54:10.815528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.815512 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9471bdaa4b35db888077c892efa437eff5e6f6f4afdab1d030bf1836cf7d299" Nov 03 20:54:10.831869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.831846 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:54:10.908081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.907975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:10.937211 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.937182 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-maas-api-tls\") pod \"4aa7c6ac-d377-450b-820b-19fb08c44a55\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " Nov 03 20:54:10.937357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.937249 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa7c6ac-d377-450b-820b-19fb08c44a55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa7c6ac-d377-450b-820b-19fb08c44a55-openshift-service-ca\") pod \"4aa7c6ac-d377-450b-820b-19fb08c44a55\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " Nov 03 20:54:10.937486 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.937437 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa7c6ac-d377-450b-820b-19fb08c44a55-kube-api-access-k8g48\" (UniqueName: \"kubernetes.io/projected/4aa7c6ac-d377-450b-820b-19fb08c44a55-kube-api-access-k8g48\") pod \"4aa7c6ac-d377-450b-820b-19fb08c44a55\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " Nov 03 20:54:10.937577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.937564 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-metrics-certs\") pod \"4aa7c6ac-d377-450b-820b-19fb08c44a55\" (UID: \"4aa7c6ac-d377-450b-820b-19fb08c44a55\") " Nov 03 20:54:10.937624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.937594 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa7c6ac-d377-450b-820b-19fb08c44a55-openshift-service-ca" pod "4aa7c6ac-d377-450b-820b-19fb08c44a55" (UID: "4aa7c6ac-d377-450b-820b-19fb08c44a55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:10.938042 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.938021 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa7c6ac-d377-450b-820b-19fb08c44a55-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:10.939341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.939313 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-maas-api-tls" pod "4aa7c6ac-d377-450b-820b-19fb08c44a55" (UID: "4aa7c6ac-d377-450b-820b-19fb08c44a55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:10.939572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.939506 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-metrics-certs" pod "4aa7c6ac-d377-450b-820b-19fb08c44a55" (UID: "4aa7c6ac-d377-450b-820b-19fb08c44a55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:10.939572 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:10.939540 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa7c6ac-d377-450b-820b-19fb08c44a55-kube-api-access-k8g48" pod "4aa7c6ac-d377-450b-820b-19fb08c44a55" (UID: "4aa7c6ac-d377-450b-820b-19fb08c44a55"). InnerVolumeSpecName "kube-api-access-k8g48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:11.039944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:11.039860 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8g48\" (UniqueName: \"kubernetes.io/projected/4aa7c6ac-d377-450b-820b-19fb08c44a55-kube-api-access-k8g48\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:11.039944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:11.039897 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:11.039944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:11.039917 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa7c6ac-d377-450b-820b-19fb08c44a55-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:11.599122 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:11.599079 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:11.822527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:11.822494 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5" Nov 03 20:54:11.849835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:11.849760 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5"] Nov 03 20:54:11.853586 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:11.853562 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bf7965-6bddd79f6f-hxfq5"] Nov 03 20:54:12.313359 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:12.313321 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:12.682933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:12.682855 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa7c6ac-d377-450b-820b-19fb08c44a55" path="/var/lib/kubelet/pods/4aa7c6ac-d377-450b-820b-19fb08c44a55/volumes" Nov 03 20:54:12.975936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:12.975848 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:13.679468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:13.674490 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:14.493867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:14.492153 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:15.199494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:15.199433 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:16.088467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:16.087889 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:16.677365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:16.677319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:16.820912 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:16.820874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:16.987520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:16.982481 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:17.158746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:17.158707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:17.533088 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:54:17.533041 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:54:17.541259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:17.541229 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:17.936905 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:17.936799 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:18.351936 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:18.351883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:18.637291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:18.637203 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:19.457472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:19.450097 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:20.191095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:20.191054 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:20.942293 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:20.942257 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:21.700752 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:21.700707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:22.418822 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:22.418788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:23.095667 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:23.095632 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:23.545380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:23.545328 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:23.681220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:23.681186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:23.825759 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:23.825658 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:24.120381 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:24.120292 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:24.677130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:24.677089 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:24.820085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:24.820051 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:24.975396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:24.975296 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:25.113252 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:25.113215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:25.337804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:25.337770 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:25.506517 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:25.505182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:25.964339 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:25.964306 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:26.270704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:26.270669 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:26.991789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:26.991738 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:27.695692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:27.695657 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:28.429539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:28.429487 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:29.159657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:29.157673 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:29.896750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:29.896703 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:30.642027 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:30.641984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:31.341473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:31.330953 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:31.962099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:31.962063 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:32.087260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:32.087206 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:32.718698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:32.718645 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:33.409562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:33.409520 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:34.134473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:34.116779 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:34.880299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:34.880265 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:35.609686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:35.609650 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:36.258957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:36.258924 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:36.867035 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:36.866972 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:37.085592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:37.085203 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:37.721097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:37.721059 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:38.581325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:38.581290 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:39.330508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:39.319332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:39.761091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:39.761063 4711 scope.go:122] "RemoveContainer" containerID="8402d2fce6d542a58342da8ee62c4f87e6f6adc6e49a4cb7571028499dd3cad4" Nov 03 20:54:39.929381 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:39.929342 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2"] Nov 03 20:54:39.930983 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:39.930948 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" podUID="b16636b8-cceb-41ef-af18-8052c9429822" containerName="maas-api" containerID="cri-o://28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489" gracePeriod=30 Nov 03 20:54:40.010825 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.010781 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:40.485401 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.485376 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:54:40.524249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.524216 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16636b8-cceb-41ef-af18-8052c9429822-kube-api-access-zck59\" (UniqueName: \"kubernetes.io/projected/b16636b8-cceb-41ef-af18-8052c9429822-kube-api-access-zck59\") pod \"b16636b8-cceb-41ef-af18-8052c9429822\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " Nov 03 20:54:40.524429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.524417 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16636b8-cceb-41ef-af18-8052c9429822-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16636b8-cceb-41ef-af18-8052c9429822-openshift-service-ca\") pod \"b16636b8-cceb-41ef-af18-8052c9429822\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " Nov 03 20:54:40.524555 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.524543 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-metrics-certs\") pod \"b16636b8-cceb-41ef-af18-8052c9429822\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " Nov 03 20:54:40.524634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.524614 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-maas-api-tls\") pod \"b16636b8-cceb-41ef-af18-8052c9429822\" (UID: \"b16636b8-cceb-41ef-af18-8052c9429822\") " Nov 03 20:54:40.524843 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.524820 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16636b8-cceb-41ef-af18-8052c9429822-openshift-service-ca" pod "b16636b8-cceb-41ef-af18-8052c9429822" (UID: "b16636b8-cceb-41ef-af18-8052c9429822"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:40.525425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.525397 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b16636b8-cceb-41ef-af18-8052c9429822-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:40.526588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.526560 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16636b8-cceb-41ef-af18-8052c9429822-kube-api-access-zck59" pod "b16636b8-cceb-41ef-af18-8052c9429822" (UID: "b16636b8-cceb-41ef-af18-8052c9429822"). InnerVolumeSpecName "kube-api-access-zck59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:40.526888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.526869 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-maas-api-tls" pod "b16636b8-cceb-41ef-af18-8052c9429822" (UID: "b16636b8-cceb-41ef-af18-8052c9429822"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:40.527142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.527125 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-metrics-certs" pod "b16636b8-cceb-41ef-af18-8052c9429822" (UID: "b16636b8-cceb-41ef-af18-8052c9429822"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:40.553372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.553339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:40.649469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.647817 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:40.649469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.647859 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b16636b8-cceb-41ef-af18-8052c9429822-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:40.649469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:40.647875 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zck59\" (UniqueName: \"kubernetes.io/projected/b16636b8-cceb-41ef-af18-8052c9429822-kube-api-access-zck59\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:41.051115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.051084 4711 generic.go:356] "Generic (PLEG): container finished" podID="b16636b8-cceb-41ef-af18-8052c9429822" containerID="28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489" exitCode=0 Nov 03 20:54:41.051541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.051125 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" Nov 03 20:54:41.051541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.051143 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" event={"ID":"b16636b8-cceb-41ef-af18-8052c9429822","Type":"ContainerDied","Data":"28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489"} Nov 03 20:54:41.051541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.051185 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2" event={"ID":"b16636b8-cceb-41ef-af18-8052c9429822","Type":"ContainerDied","Data":"381329f052f944d54baccdec8e36679c3359c4e9a11e6ff970cfc0100fb626c6"} Nov 03 20:54:41.051541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.051208 4711 scope.go:122] "RemoveContainer" containerID="28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489" Nov 03 20:54:41.061919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.061899 4711 scope.go:122] "RemoveContainer" containerID="28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489" Nov 03 20:54:41.062177 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:54:41.062155 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489\": container with ID starting with 28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489 not found: ID does not exist" containerID="28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489" Nov 03 20:54:41.062246 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.062189 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489"} err="failed to get container status \"28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489\": rpc error: code = NotFound desc = could not find container \"28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489\": container with ID starting with 28057c5e0af6b1c92037d80f617bc2be211efe0787dadf363270068c74e98489 not found: ID does not exist" Nov 03 20:54:41.072100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.072079 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2"] Nov 03 20:54:41.075804 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.075785 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d7c7fd-794f8fc9db-tgbs2"] Nov 03 20:54:41.456271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:41.456179 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:42.682405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:42.682371 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b16636b8-cceb-41ef-af18-8052c9429822" path="/var/lib/kubelet/pods/b16636b8-cceb-41ef-af18-8052c9429822/volumes" Nov 03 20:54:43.644716 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:43.644665 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:43.771848 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:43.771806 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:44.307575 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:44.307534 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:45.028782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:45.028724 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:45.726847 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:45.726811 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:46.470487 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:46.469343 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:47.092867 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:47.092832 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:47.524604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:47.524555 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:47.534005 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:54:47.533971 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:54:47.670800 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:47.670768 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:47.860071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:47.859979 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:48.080176 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:48.080143 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:48.258350 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:48.258310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:48.500245 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:48.500210 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:49.059476 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:49.058791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:49.564207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:49.564173 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:49.818944 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:49.818833 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:49.984146 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:49.984107 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:50.393566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:50.393513 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:50.572091 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:50.572038 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:50.922347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:50.922309 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:51.854193 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:51.850032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:52.698922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:52.698874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:53.354154 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:53.354122 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:53.511380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:53.511339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:53.676518 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:53.668160 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:54.537275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:54.537239 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:54.682913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:54.682877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:55.424217 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:55.416833 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:56.206325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:56.206277 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:57.036470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:57.028434 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:58.046059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:58.046021 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:58.564535 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:58.560012 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:58.830203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:58.830112 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:59.391040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:59.390999 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:59.520802 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:59.520764 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:54:59.652991 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:54:59.652878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:00.583302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:00.583267 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:00.722781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:00.722746 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:01.723203 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:01.723081 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:01.862956 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:01.862832 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:02.007522 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:02.001550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:02.459997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:02.459898 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:03.296345 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:03.296305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:04.200938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:04.200895 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:04.942057 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:04.942014 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:05.734970 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:05.734927 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:05.865623 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:05.865576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:06.521205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:06.521152 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:06.980269 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:06.980155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:07.296037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:07.296006 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:08.061256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:08.061223 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:08.317087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:08.316962 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:08.447657 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:08.447607 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:08.923962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:08.923921 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:09.064627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:09.064591 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:09.723913 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:09.723884 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:10.501256 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:10.501211 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:11.174074 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:11.174033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:11.919720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:11.919684 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:12.642470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:12.638358 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:13.258732 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:13.258694 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:13.980490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:13.979560 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:14.610714 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:14.610651 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:15.222014 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.221982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:15.308669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.308636 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp"] Nov 03 20:55:15.309540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309517 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b16636b8-cceb-41ef-af18-8052c9429822" containerName="maas-api" Nov 03 20:55:15.309540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309540 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16636b8-cceb-41ef-af18-8052c9429822" containerName="maas-api" Nov 03 20:55:15.309736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309558 4711 cpu_manager.go:414] "containerMap: removing container" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" Nov 03 20:55:15.309736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309565 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" Nov 03 20:55:15.309736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309607 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4aa7c6ac-d377-450b-820b-19fb08c44a55" containerName="maas-api" Nov 03 20:55:15.309736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309616 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa7c6ac-d377-450b-820b-19fb08c44a55" containerName="maas-api" Nov 03 20:55:15.309736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309677 4711 cpu_manager.go:414] "containerMap: removing container" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" Nov 03 20:55:15.309736 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309685 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" Nov 03 20:55:15.310039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309882 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b58e1c-d802-4a1d-9833-c2557d8128e4" containerName="istio-proxy" Nov 03 20:55:15.310039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309901 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16636b8-cceb-41ef-af18-8052c9429822" containerName="maas-api" Nov 03 20:55:15.310039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309912 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ed0c60-5fec-4a41-912b-f5f2e9ff1516" containerName="istio-proxy" Nov 03 20:55:15.310039 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.309926 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa7c6ac-d377-450b-820b-19fb08c44a55" containerName="maas-api" Nov 03 20:55:15.314062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.314033 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.328133 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.328098 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp"] Nov 03 20:55:15.330320 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.329604 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f3a4c29a-serving-cert\"" Nov 03 20:55:15.385277 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.385242 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ftr\" (UniqueName: \"kubernetes.io/projected/0f5b00b9-b90b-47de-8345-4937e5bb688c-kube-api-access-g2ftr\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.385579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.385310 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5b00b9-b90b-47de-8345-4937e5bb688c-openshift-service-ca\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.385579 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.385511 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-maas-api-tls\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.385728 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.385666 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-metrics-certs\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.388109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.388078 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:15.486616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.486534 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-maas-api-tls\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.486797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.486648 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-metrics-certs\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.486797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.486695 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ftr\" (UniqueName: \"kubernetes.io/projected/0f5b00b9-b90b-47de-8345-4937e5bb688c-kube-api-access-g2ftr\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.486797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.486742 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5b00b9-b90b-47de-8345-4937e5bb688c-openshift-service-ca\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.487315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.487295 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5b00b9-b90b-47de-8345-4937e5bb688c-openshift-service-ca\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.488962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.488935 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-maas-api-tls\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.489118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.489097 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-metrics-certs\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.498946 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.498924 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ftr\" (UniqueName: \"kubernetes.io/projected/0f5b00b9-b90b-47de-8345-4937e5bb688c-kube-api-access-g2ftr\") pod \"maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:15.535032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.534994 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:15.649030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.649003 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:16.015463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:15.997814 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp"] Nov 03 20:55:16.019684 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:55:16.018997 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5b00b9_b90b_47de_8345_4937e5bb688c.slice/crio-a0801397a81d2c1c9d96543120d71a1d790285adc3c6e3fdc091f465f103a915 WatchSource:0}: Error finding container a0801397a81d2c1c9d96543120d71a1d790285adc3c6e3fdc091f465f103a915: Status 404 returned error can't find the container with id a0801397a81d2c1c9d96543120d71a1d790285adc3c6e3fdc091f465f103a915 Nov 03 20:55:16.031310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.025025 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764"] Nov 03 20:55:16.031310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.025265 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" podUID="c281bd1d-9536-4754-8f2c-75596c7f9ac6" containerName="maas-api" containerID="cri-o://17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718" gracePeriod=30 Nov 03 20:55:16.103904 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.103865 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:16.326730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.326699 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:55:16.448960 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.448926 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c281bd1d-9536-4754-8f2c-75596c7f9ac6-kube-api-access-zft4b\" (UniqueName: \"kubernetes.io/projected/c281bd1d-9536-4754-8f2c-75596c7f9ac6-kube-api-access-zft4b\") pod \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " Nov 03 20:55:16.449255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.449234 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c281bd1d-9536-4754-8f2c-75596c7f9ac6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c281bd1d-9536-4754-8f2c-75596c7f9ac6-openshift-service-ca\") pod \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " Nov 03 20:55:16.449508 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.449491 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-metrics-certs\") pod \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " Nov 03 20:55:16.465337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.465305 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-maas-api-tls\") pod \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\" (UID: \"c281bd1d-9536-4754-8f2c-75596c7f9ac6\") " Nov 03 20:55:16.472924 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.472888 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c281bd1d-9536-4754-8f2c-75596c7f9ac6-openshift-service-ca" pod "c281bd1d-9536-4754-8f2c-75596c7f9ac6" (UID: "c281bd1d-9536-4754-8f2c-75596c7f9ac6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:16.473788 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.473731 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:16.531730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.510667 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-maas-api-tls" pod "c281bd1d-9536-4754-8f2c-75596c7f9ac6" (UID: "c281bd1d-9536-4754-8f2c-75596c7f9ac6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:16.554506 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.536761 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-metrics-certs" pod "c281bd1d-9536-4754-8f2c-75596c7f9ac6" (UID: "c281bd1d-9536-4754-8f2c-75596c7f9ac6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:16.599751 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.599673 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c281bd1d-9536-4754-8f2c-75596c7f9ac6-kube-api-access-zft4b" pod "c281bd1d-9536-4754-8f2c-75596c7f9ac6" (UID: "c281bd1d-9536-4754-8f2c-75596c7f9ac6"). InnerVolumeSpecName "kube-api-access-zft4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:16.632387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.632339 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zft4b\" (UniqueName: \"kubernetes.io/projected/c281bd1d-9536-4754-8f2c-75596c7f9ac6-kube-api-access-zft4b\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:16.632655 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.632633 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c281bd1d-9536-4754-8f2c-75596c7f9ac6-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:16.632790 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.632776 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:16.632888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.632876 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c281bd1d-9536-4754-8f2c-75596c7f9ac6-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:16.649495 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.649372 4711 generic.go:356] "Generic (PLEG): container finished" podID="c281bd1d-9536-4754-8f2c-75596c7f9ac6" containerID="17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718" exitCode=0 Nov 03 20:55:16.649883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.649558 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" Nov 03 20:55:16.651612 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.651590 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" event={"ID":"c281bd1d-9536-4754-8f2c-75596c7f9ac6","Type":"ContainerDied","Data":"17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718"} Nov 03 20:55:16.651892 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.651872 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764" event={"ID":"c281bd1d-9536-4754-8f2c-75596c7f9ac6","Type":"ContainerDied","Data":"aea22079f3f718a22612e5463cb80a523c7e7e6a512fe55c2b87f495b2f2cda3"} Nov 03 20:55:16.652018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.651911 4711 scope.go:122] "RemoveContainer" containerID="17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718" Nov 03 20:55:16.652124 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.652077 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" event={"ID":"0f5b00b9-b90b-47de-8345-4937e5bb688c","Type":"ContainerStarted","Data":"a0801397a81d2c1c9d96543120d71a1d790285adc3c6e3fdc091f465f103a915"} Nov 03 20:55:16.672248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.672232 4711 scope.go:122] "RemoveContainer" containerID="17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718" Nov 03 20:55:16.674438 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:55:16.674139 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718\": container with ID starting with 17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718 not found: ID does not exist" containerID="17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718" Nov 03 20:55:16.674438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.674200 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718"} err="failed to get container status \"17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718\": rpc error: code = NotFound desc = could not find container \"17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718\": container with ID starting with 17d1fbec99c870b8f2803746b169731a9522f1a2395ef4e820c3adbef6180718 not found: ID does not exist" Nov 03 20:55:16.688326 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.688296 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764"] Nov 03 20:55:16.692067 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:16.692038 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4d63cd-5b5c6c4ffb-74764"] Nov 03 20:55:17.284054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:17.283961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:17.533990 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:55:17.533957 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:55:17.662996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:17.662888 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" event={"ID":"0f5b00b9-b90b-47de-8345-4937e5bb688c","Type":"ContainerStarted","Data":"34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117"} Nov 03 20:55:17.662996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:17.662960 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:17.686072 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:17.686023 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" podStartSLOduration=2.03691937 podStartE2EDuration="2.686009877s" totalImagesPullingTime="649.090507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:17.683102394 +0000 UTC m=+3221.133208444" watchObservedRunningTime="2026-11-03 20:55:17.686009877 +0000 UTC m=+3221.136115922" Nov 03 20:55:17.964130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:17.964041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:18.682744 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:18.682711 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c281bd1d-9536-4754-8f2c-75596c7f9ac6" path="/var/lib/kubelet/pods/c281bd1d-9536-4754-8f2c-75596c7f9ac6/volumes" Nov 03 20:55:18.704312 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:18.704279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:19.415882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:19.415850 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:20.269408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:20.261812 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:20.853163 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:20.853113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:21.598566 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:21.598525 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:21.906851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:21.906749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:22.056033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:22.056003 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:22.324479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:22.321990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:22.458105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:22.458031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:22.796054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:22.796014 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:23.338929 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:23.338895 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:23.677061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:23.676994 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:55:23.698734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:23.698674 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:24.035430 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:24.035395 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:24.910054 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:24.909696 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:25.598962 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:25.598929 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:26.398037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:26.397998 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:27.093851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:27.093816 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:27.785650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:27.785620 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:28.623546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:28.623506 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:29.486400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:29.486364 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:29.795304 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:29.795270 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq"] Nov 03 20:55:29.902755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:29.902720 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:30.057175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:30.045019 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:30.183989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:30.183954 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:30.622469 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:30.621548 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:31.501329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:31.501291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:32.197141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:32.197100 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:33.054246 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:33.054204 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:33.717900 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:33.717863 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:34.397919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:34.397879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:34.795852 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:34.795813 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" podUID="cc0e4e8a-eef2-428c-8b18-2507a7dc7a64" containerName="payload-processing" containerID="cri-o://2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5" gracePeriod=25 Nov 03 20:55:35.137876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.137836 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:35.147977 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.147950 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:55:35.188976 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.188943 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-kube-api-access-vg4kx\" (UniqueName: \"kubernetes.io/projected/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-kube-api-access-vg4kx\") pod \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\" (UID: \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\") " Nov 03 20:55:35.189179 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.189124 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-plugins-config-volume\") pod \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\" (UID: \"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64\") " Nov 03 20:55:35.189610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.189581 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-plugins-config-volume" pod "cc0e4e8a-eef2-428c-8b18-2507a7dc7a64" (UID: "cc0e4e8a-eef2-428c-8b18-2507a7dc7a64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:35.191668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.191641 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-kube-api-access-vg4kx" pod "cc0e4e8a-eef2-428c-8b18-2507a7dc7a64" (UID: "cc0e4e8a-eef2-428c-8b18-2507a7dc7a64"). InnerVolumeSpecName "kube-api-access-vg4kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:35.291688 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.291657 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4kx\" (UniqueName: \"kubernetes.io/projected/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-kube-api-access-vg4kx\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:35.291688 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.291684 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:35.819286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.819254 4711 generic.go:356] "Generic (PLEG): container finished" podID="cc0e4e8a-eef2-428c-8b18-2507a7dc7a64" containerID="2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5" exitCode=0 Nov 03 20:55:35.819762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.819476 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" event={"ID":"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64","Type":"ContainerDied","Data":"2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5"} Nov 03 20:55:35.819762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.819523 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" event={"ID":"cc0e4e8a-eef2-428c-8b18-2507a7dc7a64","Type":"ContainerDied","Data":"a5738fdbe07352e2073766532c03b0cde1a49d6a24aa0c60129220a866bfe3a5"} Nov 03 20:55:35.819762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.819545 4711 scope.go:122] "RemoveContainer" containerID="2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5" Nov 03 20:55:35.819762 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.819675 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq" Nov 03 20:55:35.834902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.834879 4711 scope.go:122] "RemoveContainer" containerID="2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5" Nov 03 20:55:35.835215 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:55:35.835195 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5\": container with ID starting with 2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5 not found: ID does not exist" containerID="2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5" Nov 03 20:55:35.835291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.835223 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5"} err="failed to get container status \"2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5\": rpc error: code = NotFound desc = could not find container \"2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5\": container with ID starting with 2dfd4a3cae256b4ca578e53c86c95bafff5ca84faf119354c8077d8a6e8a00c5 not found: ID does not exist" Nov 03 20:55:35.849853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.849828 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq"] Nov 03 20:55:35.855088 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:35.855064 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-519a78-f84b65d4-dqlbq"] Nov 03 20:55:36.025687 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:36.025650 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:36.682190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:36.682161 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0e4e8a-eef2-428c-8b18-2507a7dc7a64" path="/var/lib/kubelet/pods/cc0e4e8a-eef2-428c-8b18-2507a7dc7a64/volumes" Nov 03 20:55:36.823467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:36.822939 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:37.214151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:37.214116 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:37.795798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:37.795753 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:38.811470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:38.807868 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:39.135683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:39.135595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:39.617429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:39.617390 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:40.012858 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:40.012826 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:40.143239 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:40.143204 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:40.572468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:40.563111 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:40.698440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:40.698405 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:41.271954 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:41.271917 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:41.406169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:41.406133 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:42.040730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:42.040682 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:42.570034 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:42.570001 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf"] Nov 03 20:55:42.570504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:42.570239 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" podUID="cc40b733-4aaf-445c-a516-53810817296c" containerName="maas-api" containerID="cri-o://c581e51a29107feaa6db5980ba4d057424b13c3be040b3cd53474657b50abbf6" gracePeriod=30 Nov 03 20:55:42.655604 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:42.655570 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:43.043365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.043216 4711 generic.go:356] "Generic (PLEG): container finished" podID="cc40b733-4aaf-445c-a516-53810817296c" containerID="c581e51a29107feaa6db5980ba4d057424b13c3be040b3cd53474657b50abbf6" exitCode=0 Nov 03 20:55:43.043365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.043297 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" event={"ID":"cc40b733-4aaf-445c-a516-53810817296c","Type":"ContainerDied","Data":"c581e51a29107feaa6db5980ba4d057424b13c3be040b3cd53474657b50abbf6"} Nov 03 20:55:43.047248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.047219 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:55:43.165003 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.164970 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:43.227142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.226618 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc40b733-4aaf-445c-a516-53810817296c-kube-api-access-692q7\" (UniqueName: \"kubernetes.io/projected/cc40b733-4aaf-445c-a516-53810817296c-kube-api-access-692q7\") pod \"cc40b733-4aaf-445c-a516-53810817296c\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " Nov 03 20:55:43.227142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.226757 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc40b733-4aaf-445c-a516-53810817296c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40b733-4aaf-445c-a516-53810817296c-openshift-service-ca\") pod \"cc40b733-4aaf-445c-a516-53810817296c\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " Nov 03 20:55:43.227142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.226815 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-metrics-certs\") pod \"cc40b733-4aaf-445c-a516-53810817296c\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " Nov 03 20:55:43.227142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.227003 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-maas-api-tls\") pod \"cc40b733-4aaf-445c-a516-53810817296c\" (UID: \"cc40b733-4aaf-445c-a516-53810817296c\") " Nov 03 20:55:43.235023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.231370 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc40b733-4aaf-445c-a516-53810817296c-openshift-service-ca" pod "cc40b733-4aaf-445c-a516-53810817296c" (UID: "cc40b733-4aaf-445c-a516-53810817296c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:43.264703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.264670 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-maas-api-tls" pod "cc40b733-4aaf-445c-a516-53810817296c" (UID: "cc40b733-4aaf-445c-a516-53810817296c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:43.266360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.266333 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-metrics-certs" pod "cc40b733-4aaf-445c-a516-53810817296c" (UID: "cc40b733-4aaf-445c-a516-53810817296c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:43.266691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.266671 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc40b733-4aaf-445c-a516-53810817296c-kube-api-access-692q7" pod "cc40b733-4aaf-445c-a516-53810817296c" (UID: "cc40b733-4aaf-445c-a516-53810817296c"). InnerVolumeSpecName "kube-api-access-692q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:43.329797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.329763 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-692q7\" (UniqueName: \"kubernetes.io/projected/cc40b733-4aaf-445c-a516-53810817296c-kube-api-access-692q7\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:43.329797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.329792 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc40b733-4aaf-445c-a516-53810817296c-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:43.329797 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.329804 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:43.330022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.329813 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc40b733-4aaf-445c-a516-53810817296c-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:43.807215 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.807182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:43.944539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:43.944422 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:44.051683 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.051653 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" Nov 03 20:55:44.051889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.051658 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf" event={"ID":"cc40b733-4aaf-445c-a516-53810817296c","Type":"ContainerDied","Data":"c28bee3f1569e5a117e1dffa08ef7e9b45cd4ef4e424f3c9b6b82f907087804b"} Nov 03 20:55:44.051889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.051787 4711 scope.go:122] "RemoveContainer" containerID="c581e51a29107feaa6db5980ba4d057424b13c3be040b3cd53474657b50abbf6" Nov 03 20:55:44.133951 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.133915 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf"] Nov 03 20:55:44.144594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.144560 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-519a78-8468dc4cb4-qwsjf"] Nov 03 20:55:44.397671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.397634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:44.689289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.689204 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc40b733-4aaf-445c-a516-53810817296c" path="/var/lib/kubelet/pods/cc40b733-4aaf-445c-a516-53810817296c/volumes" Nov 03 20:55:44.693973 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.693940 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h"] Nov 03 20:55:44.694492 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.694462 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" containerID="cri-o://f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16" gracePeriod=30 Nov 03 20:55:44.758125 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.758092 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:44.931467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:44.923401 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:45.660791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:45.660759 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:45.797136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:45.797096 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:45.934378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:45.934283 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:46.087782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:46.087745 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:46.592538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:46.592314 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:47.306496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:47.306464 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:47.467471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:47.460983 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:47.535404 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:55:47.535361 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:55:48.101565 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:48.101458 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:48.756386 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:48.756343 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:49.545471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:49.539360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:49.987299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:49.987269 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:55:50.070505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070395 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-podinfo\") pod \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " Nov 03 20:55:50.070505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070490 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-envoy\") pod \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " Nov 03 20:55:50.070730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070540 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-credential-socket\") pod \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " Nov 03 20:55:50.070730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070559 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-data\") pod \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " Nov 03 20:55:50.070730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070634 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-token\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-token\") pod \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " Nov 03 20:55:50.070730 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070700 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istiod-ca-cert\") pod \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " Nov 03 20:55:50.070966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070839 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-kube-api-access-dq9k2\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-kube-api-access-dq9k2\") pod \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " Nov 03 20:55:50.070966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070850 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-credential-socket" pod "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" (UID: "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:50.070966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070866 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-data" pod "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" (UID: "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:50.070966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070884 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-certs\") pod \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " Nov 03 20:55:50.070966 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.070932 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-socket\") pod \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\" (UID: \"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9\") " Nov 03 20:55:50.071254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.071075 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-certs" pod "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" (UID: "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:50.071254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.071161 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istiod-ca-cert" pod "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" (UID: "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:50.071369 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.071351 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-socket" pod "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" (UID: "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:50.071856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.071829 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.071984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.071859 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.071984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.071875 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.071984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.071891 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.071984 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.071904 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.072827 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.072795 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-envoy" pod "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" (UID: "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:50.073009 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.072991 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-kube-api-access-dq9k2" pod "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" (UID: "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9"). InnerVolumeSpecName "kube-api-access-dq9k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:50.073097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.073077 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-podinfo" pod "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" (UID: "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:55:50.073152 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.073107 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-token" pod "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" (UID: "77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:50.111557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.111522 4711 generic.go:356] "Generic (PLEG): container finished" podID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerID="f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16" exitCode=0 Nov 03 20:55:50.111725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.111602 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" event={"ID":"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9","Type":"ContainerDied","Data":"f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16"} Nov 03 20:55:50.111725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.111663 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" event={"ID":"77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9","Type":"ContainerDied","Data":"f9c45c319fad731e45e18f7d02b6e0f5ab87e1191669733bdfc4b5255e98a403"} Nov 03 20:55:50.111725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.111608 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h" Nov 03 20:55:50.111725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.111686 4711 scope.go:122] "RemoveContainer" containerID="f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16" Nov 03 20:55:50.123131 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.123110 4711 scope.go:122] "RemoveContainer" containerID="206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c" Nov 03 20:55:50.134207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.134184 4711 scope.go:122] "RemoveContainer" containerID="f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16" Nov 03 20:55:50.134511 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:55:50.134487 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16\": container with ID starting with f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16 not found: ID does not exist" containerID="f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16" Nov 03 20:55:50.134556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.134525 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16"} err="failed to get container status \"f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16\": rpc error: code = NotFound desc = could not find container \"f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16\": container with ID starting with f5e9f1fa0f1d3b7ca07b0051406e91c7fa08b4d7ae8453c559814f3feeeade16 not found: ID does not exist" Nov 03 20:55:50.134556 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.134551 4711 scope.go:122] "RemoveContainer" containerID="206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c" Nov 03 20:55:50.134829 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:55:50.134808 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c\": container with ID starting with 206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c not found: ID does not exist" containerID="206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c" Nov 03 20:55:50.134872 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.134836 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c"} err="failed to get container status \"206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c\": rpc error: code = NotFound desc = could not find container \"206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c\": container with ID starting with 206b71ae4fefd54f3834d9b2a02ca1f191b2696713483025ccc7a3173decaf6c not found: ID does not exist" Nov 03 20:55:50.137325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.137305 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h"] Nov 03 20:55:50.140997 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.140976 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-519a78-openshift-default-5d4d979b7f-h8w2h"] Nov 03 20:55:50.173392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.173366 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.173392 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.173392 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq9k2\" (UniqueName: \"kubernetes.io/projected/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-kube-api-access-dq9k2\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.173601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.173402 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.173601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.173413 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:50.392869 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.392771 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:50.681372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:50.681299 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" path="/var/lib/kubelet/pods/77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9/volumes" Nov 03 20:55:51.151149 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:51.151113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:51.941387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:51.941351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:52.486323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:52.486288 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:52.616166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:52.616125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:52.758491 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:52.757721 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:53.505664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:53.505621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:54.035820 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:54.035772 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:54.527105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:54.527071 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:55.330540 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:55.330499 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:56.132051 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:56.131955 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:56.919855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:56.919814 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:57.712616 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:57.712582 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:58.406262 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:58.406228 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:55:59.198560 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:55:59.198522 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:00.917663 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:00.917623 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:01.422130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:01.422084 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:02.186619 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:02.186577 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:03.024474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:03.024419 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:03.543460 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:03.543421 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2"] Nov 03 20:56:03.544128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:03.543665 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" containerID="cri-o://b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6" gracePeriod=30 Nov 03 20:56:03.868093 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:03.867983 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:04.608512 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:04.608472 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:05.293961 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:05.293925 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:06.080087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:06.080049 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:06.674792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:06.674758 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:56:06.674992 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:06.674832 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:56:06.853819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:06.853785 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:07.644109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:07.644066 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:08.296482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.296408 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:08.834366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.834343 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:56:08.928554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928520 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-kube-api-access-twbmm\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-kube-api-access-twbmm\") pod \"3e3f57ad-c030-4739-8723-f728de7a129a\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " Nov 03 20:56:08.928554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928557 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-socket\") pod \"3e3f57ad-c030-4739-8723-f728de7a129a\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " Nov 03 20:56:08.928844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928589 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-data\") pod \"3e3f57ad-c030-4739-8723-f728de7a129a\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " Nov 03 20:56:08.928844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928623 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3f57ad-c030-4739-8723-f728de7a129a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3f57ad-c030-4739-8723-f728de7a129a-istiod-ca-cert\") pod \"3e3f57ad-c030-4739-8723-f728de7a129a\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " Nov 03 20:56:08.928844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928642 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-envoy\") pod \"3e3f57ad-c030-4739-8723-f728de7a129a\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " Nov 03 20:56:08.928844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928775 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e3f57ad-c030-4739-8723-f728de7a129a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3f57ad-c030-4739-8723-f728de7a129a-istio-podinfo\") pod \"3e3f57ad-c030-4739-8723-f728de7a129a\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " Nov 03 20:56:08.928844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928824 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-credential-socket\") pod \"3e3f57ad-c030-4739-8723-f728de7a129a\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " Nov 03 20:56:08.929130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928861 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-socket" pod "3e3f57ad-c030-4739-8723-f728de7a129a" (UID: "3e3f57ad-c030-4739-8723-f728de7a129a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.929130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928918 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-istio-token\") pod \"3e3f57ad-c030-4739-8723-f728de7a129a\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " Nov 03 20:56:08.929130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.928949 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-certs\") pod \"3e3f57ad-c030-4739-8723-f728de7a129a\" (UID: \"3e3f57ad-c030-4739-8723-f728de7a129a\") " Nov 03 20:56:08.929130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.929017 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-data" pod "3e3f57ad-c030-4739-8723-f728de7a129a" (UID: "3e3f57ad-c030-4739-8723-f728de7a129a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.929355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.929130 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3f57ad-c030-4739-8723-f728de7a129a-istiod-ca-cert" pod "3e3f57ad-c030-4739-8723-f728de7a129a" (UID: "3e3f57ad-c030-4739-8723-f728de7a129a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:08.929355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.929153 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-credential-socket" pod "3e3f57ad-c030-4739-8723-f728de7a129a" (UID: "3e3f57ad-c030-4739-8723-f728de7a129a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.929355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.929308 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-certs" pod "3e3f57ad-c030-4739-8723-f728de7a129a" (UID: "3e3f57ad-c030-4739-8723-f728de7a129a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.929714 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.929689 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.929811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.929720 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.929811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.929735 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.929811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.929751 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.929811 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.929765 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e3f57ad-c030-4739-8723-f728de7a129a-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:08.931019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.930996 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-envoy" pod "3e3f57ad-c030-4739-8723-f728de7a129a" (UID: "3e3f57ad-c030-4739-8723-f728de7a129a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:08.931141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.931084 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-istio-token" pod "3e3f57ad-c030-4739-8723-f728de7a129a" (UID: "3e3f57ad-c030-4739-8723-f728de7a129a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:08.931141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.931089 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e3f57ad-c030-4739-8723-f728de7a129a-istio-podinfo" pod "3e3f57ad-c030-4739-8723-f728de7a129a" (UID: "3e3f57ad-c030-4739-8723-f728de7a129a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:56:08.931228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:08.931169 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-kube-api-access-twbmm" pod "3e3f57ad-c030-4739-8723-f728de7a129a" (UID: "3e3f57ad-c030-4739-8723-f728de7a129a"). InnerVolumeSpecName "kube-api-access-twbmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:09.031232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.031199 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:09.031232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.031229 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twbmm\" (UniqueName: \"kubernetes.io/projected/3e3f57ad-c030-4739-8723-f728de7a129a-kube-api-access-twbmm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:09.031232 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.031239 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e3f57ad-c030-4739-8723-f728de7a129a-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:09.031529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.031251 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e3f57ad-c030-4739-8723-f728de7a129a-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:09.094819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.094720 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:09.407396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.407316 4711 generic.go:356] "Generic (PLEG): container finished" podID="3e3f57ad-c030-4739-8723-f728de7a129a" containerID="b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6" exitCode=0 Nov 03 20:56:09.407396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.407380 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" Nov 03 20:56:09.407606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.407397 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" event={"ID":"3e3f57ad-c030-4739-8723-f728de7a129a","Type":"ContainerDied","Data":"b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6"} Nov 03 20:56:09.407606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.407474 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2" event={"ID":"3e3f57ad-c030-4739-8723-f728de7a129a","Type":"ContainerDied","Data":"acea8506e17690ba845e7bd0fe24d775d03ea617832ffb65c87fdccca2eee043"} Nov 03 20:56:09.407606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.407486 4711 scope.go:122] "RemoveContainer" containerID="b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6" Nov 03 20:56:09.421425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.421408 4711 scope.go:122] "RemoveContainer" containerID="b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6" Nov 03 20:56:09.421770 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:56:09.421748 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6\": container with ID starting with b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6 not found: ID does not exist" containerID="b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6" Nov 03 20:56:09.421844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.421782 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6"} err="failed to get container status \"b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6\": rpc error: code = NotFound desc = could not find container \"b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6\": container with ID starting with b090b67a8c6ed45ccdfd160244df7e6850b575c9a43d867571c92fdb50b0b6c6 not found: ID does not exist" Nov 03 20:56:09.438081 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.438054 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2"] Nov 03 20:56:09.441701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.441677 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-vjdk2"] Nov 03 20:56:09.891036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:09.890998 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:10.036828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:10.036793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:10.681842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:10.681809 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" path="/var/lib/kubelet/pods/3e3f57ad-c030-4739-8723-f728de7a129a/volumes" Nov 03 20:56:10.700019 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:10.699980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:11.443910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:11.443862 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:12.190744 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:12.190710 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:13.018085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:13.018046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:13.793685 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:13.793642 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:14.921638 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:14.921600 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:15.599275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:15.599239 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:16.383501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:16.383426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:17.103037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:17.102991 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:17.555566 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:56:17.555523 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:56:17.745304 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:17.745268 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:17.889234 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:17.885390 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:18.524032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:18.523998 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:19.317653 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:19.315206 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:20.039753 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:20.039718 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:20.369989 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:20.369896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:20.579186 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:20.579142 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:20.920412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:20.920348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:21.644471 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:21.631573 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:22.387411 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:22.387370 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:23.117318 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:23.117286 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:23.790354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:23.790321 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:24.407889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:24.407855 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:24.551196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:24.551156 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:25.293750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:25.293711 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:26.055264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:26.055227 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:26.694289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:26.694258 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:27.353915 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:27.353867 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:28.119652 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:28.119612 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:28.797775 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:28.797737 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:29.491646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:29.491610 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:30.225573 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:30.225538 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:30.906735 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:30.906695 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:31.692567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:31.692533 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:32.416975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:32.416939 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:33.093903 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:33.093869 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:33.784710 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:33.784672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:34.601395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:34.601356 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:35.399906 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:35.399873 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:36.608479 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:36.605090 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:37.593589 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:37.593323 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:37.756467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:37.755607 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:56:37.778468 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:37.775543 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 20:56:37.783462 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:37.783422 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:56:37.858658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:37.857661 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 20:56:38.488470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:38.487777 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:39.133729 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:39.133696 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:40.006473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:40.003188 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:40.707285 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:40.707249 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:41.359768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:41.359737 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:42.029020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:42.028976 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:42.717830 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:42.717793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:43.499600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:43.499568 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:44.198731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:44.198693 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:44.924514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:44.924473 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:45.608701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:45.608663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:46.391953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:46.391918 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:47.200658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:47.200622 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:47.534032 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:56:47.533992 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:56:47.859691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:47.859608 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:48.513634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:48.513596 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:49.397159 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:49.397113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:50.096353 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:50.096313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:50.755439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:50.755405 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:51.605423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:51.605385 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:52.315805 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:52.315769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:52.999428 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:52.999370 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:53.862537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:53.857094 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:54.583082 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:54.583048 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:55.416940 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:55.416882 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:56.107943 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:56.107904 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:56.804292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:56.804250 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:57.608908 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:57.608874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:58.312755 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:58.312718 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:58.991061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:58.991023 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:59.546106 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:59.546070 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:56:59.715763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:56:59.715722 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:00.536784 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:00.536719 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:00.787845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:00.787756 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:01.186496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:01.186373 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:01.914978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:01.914947 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:02.677690 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:02.677656 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:03.025581 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:03.025548 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt"] Nov 03 20:57:03.026036 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:03.025773 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" containerID="cri-o://924a4ed0b66659cffc6980a9f4c501723428da276c67e5287f16fd86bc183cac" gracePeriod=30 Nov 03 20:57:03.410856 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:03.410738 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:04.113546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:04.113510 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:04.797213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:04.797174 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:05.530274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:05.530238 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:06.310592 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:06.310557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:07.114228 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:07.114193 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:07.872481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:07.872417 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-929nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:57:07.872792 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:07.872759 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:57:07.879189 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:07.879156 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:08.166006 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.165963 4711 generic.go:356] "Generic (PLEG): container finished" podID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerID="924a4ed0b66659cffc6980a9f4c501723428da276c67e5287f16fd86bc183cac" exitCode=0 Nov 03 20:57:08.166361 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.166013 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" event={"ID":"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9","Type":"ContainerDied","Data":"924a4ed0b66659cffc6980a9f4c501723428da276c67e5287f16fd86bc183cac"} Nov 03 20:57:08.339423 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.339399 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:57:08.479500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.478277 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-podinfo\") pod \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " Nov 03 20:57:08.479500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.478362 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-kube-api-access-4stdh\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-kube-api-access-4stdh\") pod \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " Nov 03 20:57:08.479500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.478499 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-envoy\") pod \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " Nov 03 20:57:08.479500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.478595 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-data\") pod \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " Nov 03 20:57:08.479500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.478681 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-token\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-token\") pod \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " Nov 03 20:57:08.479500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.478743 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istiod-ca-cert\") pod \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " Nov 03 20:57:08.479500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.478862 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-credential-socket\") pod \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " Nov 03 20:57:08.479500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.478967 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-certs\") pod \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " Nov 03 20:57:08.479500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.479000 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-socket\") pod \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\" (UID: \"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9\") " Nov 03 20:57:08.483707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.483673 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-data" pod "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" (UID: "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:08.483707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.483678 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-socket" pod "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" (UID: "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:08.484801 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.484777 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.485037 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.485009 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-credential-socket" pod "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" (UID: "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:08.485142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.485013 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istiod-ca-cert" pod "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" (UID: "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:08.485142 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.485018 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-certs" pod "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" (UID: "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:08.486219 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.486192 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-token" pod "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" (UID: "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:08.486863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.486837 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-podinfo" pod "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" (UID: "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:08.487022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.486999 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-envoy" pod "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" (UID: "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:08.487097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.487081 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-kube-api-access-4stdh" pod "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" (UID: "85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9"). InnerVolumeSpecName "kube-api-access-4stdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:08.586341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.586311 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.586341 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.586339 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.586538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.586351 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.586538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.586360 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.586538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.586370 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4stdh\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-kube-api-access-4stdh\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.586538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.586378 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.586538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.586387 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:08.586538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:08.586397 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:09.010793 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.010761 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:09.174098 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.174069 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" Nov 03 20:57:09.174554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.174097 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt" event={"ID":"85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9","Type":"ContainerDied","Data":"e702b6ec75554b6cd0e07caa703889206220f2f36d42a7301028a55fafa82bd9"} Nov 03 20:57:09.174554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.174163 4711 scope.go:122] "RemoveContainer" containerID="924a4ed0b66659cffc6980a9f4c501723428da276c67e5287f16fd86bc183cac" Nov 03 20:57:09.198839 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.198811 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt"] Nov 03 20:57:09.205229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.205207 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-929nt"] Nov 03 20:57:09.484812 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.484727 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs"] Nov 03 20:57:09.484967 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.484931 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" podUID="11439f87-11d4-4015-853c-b5d587d54f6d" containerName="discovery" containerID="cri-o://8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64" gracePeriod=30 Nov 03 20:57:09.781302 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.781279 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:57:09.924085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.924049 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-kube-api-access-cltwb\") pod \"11439f87-11d4-4015-853c-b5d587d54f6d\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " Nov 03 20:57:09.924274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.924196 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-dns-cert\") pod \"11439f87-11d4-4015-853c-b5d587d54f6d\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " Nov 03 20:57:09.924274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.924248 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11439f87-11d4-4015-853c-b5d587d54f6d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11439f87-11d4-4015-853c-b5d587d54f6d-local-certs\") pod \"11439f87-11d4-4015-853c-b5d587d54f6d\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " Nov 03 20:57:09.924421 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.924289 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-ca-configmap\") pod \"11439f87-11d4-4015-853c-b5d587d54f6d\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " Nov 03 20:57:09.924421 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.924376 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-kubeconfig\") pod \"11439f87-11d4-4015-853c-b5d587d54f6d\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " Nov 03 20:57:09.924561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.924478 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-cacerts\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-cacerts\") pod \"11439f87-11d4-4015-853c-b5d587d54f6d\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " Nov 03 20:57:09.924561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.924552 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-istio-token\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-istio-token\") pod \"11439f87-11d4-4015-853c-b5d587d54f6d\" (UID: \"11439f87-11d4-4015-853c-b5d587d54f6d\") " Nov 03 20:57:09.924791 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.924760 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-ca-configmap" pod "11439f87-11d4-4015-853c-b5d587d54f6d" (UID: "11439f87-11d4-4015-853c-b5d587d54f6d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:09.925358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.925334 4711 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-ca-configmap\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:09.927013 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.926937 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11439f87-11d4-4015-853c-b5d587d54f6d-local-certs" pod "11439f87-11d4-4015-853c-b5d587d54f6d" (UID: "11439f87-11d4-4015-853c-b5d587d54f6d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:09.927249 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.927226 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-kubeconfig" pod "11439f87-11d4-4015-853c-b5d587d54f6d" (UID: "11439f87-11d4-4015-853c-b5d587d54f6d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:09.927347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.927322 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-kube-api-access-cltwb" pod "11439f87-11d4-4015-853c-b5d587d54f6d" (UID: "11439f87-11d4-4015-853c-b5d587d54f6d"). InnerVolumeSpecName "kube-api-access-cltwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:09.927395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.927384 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-istio-token" pod "11439f87-11d4-4015-853c-b5d587d54f6d" (UID: "11439f87-11d4-4015-853c-b5d587d54f6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:09.927523 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.927426 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-dns-cert" pod "11439f87-11d4-4015-853c-b5d587d54f6d" (UID: "11439f87-11d4-4015-853c-b5d587d54f6d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:09.927902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.927879 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-cacerts" pod "11439f87-11d4-4015-853c-b5d587d54f6d" (UID: "11439f87-11d4-4015-853c-b5d587d54f6d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:09.977501 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:09.962829 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:10.026773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.026739 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:10.026773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.026770 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cltwb\" (UniqueName: \"kubernetes.io/projected/11439f87-11d4-4015-853c-b5d587d54f6d-kube-api-access-cltwb\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:10.026773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.026781 4711 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-csr-dns-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:10.027028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.026790 4711 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/11439f87-11d4-4015-853c-b5d587d54f6d-local-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:10.027028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.026800 4711 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-istio-kubeconfig\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:10.027028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.026809 4711 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/11439f87-11d4-4015-853c-b5d587d54f6d-cacerts\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:10.182108 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.182076 4711 generic.go:356] "Generic (PLEG): container finished" podID="11439f87-11d4-4015-853c-b5d587d54f6d" containerID="8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64" exitCode=0 Nov 03 20:57:10.182595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.182146 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" Nov 03 20:57:10.182595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.182165 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" event={"ID":"11439f87-11d4-4015-853c-b5d587d54f6d","Type":"ContainerDied","Data":"8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64"} Nov 03 20:57:10.182595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.182253 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs" event={"ID":"11439f87-11d4-4015-853c-b5d587d54f6d","Type":"ContainerDied","Data":"6a530555d62ada599a8d0218f07544fda08f348d9001c43dc4bf293dafe659df"} Nov 03 20:57:10.182595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.182279 4711 scope.go:122] "RemoveContainer" containerID="8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64" Nov 03 20:57:10.198243 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.198223 4711 scope.go:122] "RemoveContainer" containerID="8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64" Nov 03 20:57:10.198577 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:57:10.198550 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64\": container with ID starting with 8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64 not found: ID does not exist" containerID="8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64" Nov 03 20:57:10.198662 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.198585 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64"} err="failed to get container status \"8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64\": rpc error: code = NotFound desc = could not find container \"8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64\": container with ID starting with 8b6617c2b8ede892a3076d8ebd0354b660fd9d235ebfcc3f3444845130e6ed64 not found: ID does not exist" Nov 03 20:57:10.256417 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.256382 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs"] Nov 03 20:57:10.279408 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.279345 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x8ffs"] Nov 03 20:57:10.681870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.681794 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11439f87-11d4-4015-853c-b5d587d54f6d" path="/var/lib/kubelet/pods/11439f87-11d4-4015-853c-b5d587d54f6d/volumes" Nov 03 20:57:10.682294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.682279 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" path="/var/lib/kubelet/pods/85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9/volumes" Nov 03 20:57:10.822391 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:10.822359 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:11.396883 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:11.396843 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:11.649713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:11.649633 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:12.598113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:12.598077 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:13.428965 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:13.428920 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:14.307882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:14.307838 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:15.116816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:15.116764 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:15.804691 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:15.804556 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:16.498642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:16.498516 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:17.355784 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:17.355742 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:17.498933 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:17.498892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:17.532924 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:57:17.532881 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:57:18.227289 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:18.227255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:18.749145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:18.749113 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:18.881851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:18.881820 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:19.023161 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:19.023127 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:19.856987 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:19.856948 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:20.498033 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:20.497990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:21.205049 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:21.205012 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:21.920727 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:21.920679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:22.397498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.397464 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp"] Nov 03 20:57:22.397949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.397718 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" podUID="0f5b00b9-b90b-47de-8345-4937e5bb688c" containerName="maas-api" containerID="cri-o://34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117" gracePeriod=30 Nov 03 20:57:22.482071 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.482036 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:22.788418 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.765987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:22.864372 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.864346 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:57:22.915939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.915894 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-maas-api-tls\") pod \"0f5b00b9-b90b-47de-8345-4937e5bb688c\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " Nov 03 20:57:22.916111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.915945 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f5b00b9-b90b-47de-8345-4937e5bb688c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5b00b9-b90b-47de-8345-4937e5bb688c-openshift-service-ca\") pod \"0f5b00b9-b90b-47de-8345-4937e5bb688c\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " Nov 03 20:57:22.916111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.915996 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5b00b9-b90b-47de-8345-4937e5bb688c-kube-api-access-g2ftr\" (UniqueName: \"kubernetes.io/projected/0f5b00b9-b90b-47de-8345-4937e5bb688c-kube-api-access-g2ftr\") pod \"0f5b00b9-b90b-47de-8345-4937e5bb688c\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " Nov 03 20:57:22.916184 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.916113 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-metrics-certs\") pod \"0f5b00b9-b90b-47de-8345-4937e5bb688c\" (UID: \"0f5b00b9-b90b-47de-8345-4937e5bb688c\") " Nov 03 20:57:22.916877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.916631 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5b00b9-b90b-47de-8345-4937e5bb688c-openshift-service-ca" pod "0f5b00b9-b90b-47de-8345-4937e5bb688c" (UID: "0f5b00b9-b90b-47de-8345-4937e5bb688c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:22.916877 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.916849 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f5b00b9-b90b-47de-8345-4937e5bb688c-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:22.918782 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.918760 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-metrics-certs" pod "0f5b00b9-b90b-47de-8345-4937e5bb688c" (UID: "0f5b00b9-b90b-47de-8345-4937e5bb688c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:22.920787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.920762 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-maas-api-tls" pod "0f5b00b9-b90b-47de-8345-4937e5bb688c" (UID: "0f5b00b9-b90b-47de-8345-4937e5bb688c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:22.922030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:22.922006 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5b00b9-b90b-47de-8345-4937e5bb688c-kube-api-access-g2ftr" pod "0f5b00b9-b90b-47de-8345-4937e5bb688c" (UID: "0f5b00b9-b90b-47de-8345-4937e5bb688c"). InnerVolumeSpecName "kube-api-access-g2ftr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:23.018378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.018345 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.018378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.018373 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2ftr\" (UniqueName: \"kubernetes.io/projected/0f5b00b9-b90b-47de-8345-4937e5bb688c-kube-api-access-g2ftr\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.018603 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.018389 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f5b00b9-b90b-47de-8345-4937e5bb688c-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.294930 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.294893 4711 generic.go:356] "Generic (PLEG): container finished" podID="0f5b00b9-b90b-47de-8345-4937e5bb688c" containerID="34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117" exitCode=0 Nov 03 20:57:23.295100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.294938 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" event={"ID":"0f5b00b9-b90b-47de-8345-4937e5bb688c","Type":"ContainerDied","Data":"34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117"} Nov 03 20:57:23.295100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.294945 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" Nov 03 20:57:23.295100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.294992 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp" event={"ID":"0f5b00b9-b90b-47de-8345-4937e5bb688c","Type":"ContainerDied","Data":"a0801397a81d2c1c9d96543120d71a1d790285adc3c6e3fdc091f465f103a915"} Nov 03 20:57:23.295100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.295022 4711 scope.go:122] "RemoveContainer" containerID="34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117" Nov 03 20:57:23.306642 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.306623 4711 scope.go:122] "RemoveContainer" containerID="34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117" Nov 03 20:57:23.306893 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:57:23.306867 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117\": container with ID starting with 34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117 not found: ID does not exist" containerID="34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117" Nov 03 20:57:23.306995 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.306897 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117"} err="failed to get container status \"34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117\": rpc error: code = NotFound desc = could not find container \"34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117\": container with ID starting with 34947c9d04fabe57c28a6e21a0ee0c4405cf7b8f06b8f52504b2257b9a1f3117 not found: ID does not exist" Nov 03 20:57:23.320075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.320047 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp"] Nov 03 20:57:23.327836 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.327802 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a4c29a-6c4c55bb8c-8rdjp"] Nov 03 20:57:23.566047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:23.565956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:24.394325 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:24.394274 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:24.680853 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:24.680778 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5b00b9-b90b-47de-8345-4937e5bb688c" path="/var/lib/kubelet/pods/0f5b00b9-b90b-47de-8345-4937e5bb688c/volumes" Nov 03 20:57:25.047366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:25.047338 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:25.279922 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:25.279889 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:25.406062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:25.405982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:25.936173 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:25.936139 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:26.080044 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:26.080012 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:26.274136 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:26.274104 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:26.729968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:26.729881 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:27.117376 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.117306 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:27.478115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.478020 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j"] Nov 03 20:57:27.479062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479033 4711 cpu_manager.go:414] "containerMap: removing container" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" Nov 03 20:57:27.479062 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479060 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479079 4711 cpu_manager.go:414] "containerMap: removing container" podUID="11439f87-11d4-4015-853c-b5d587d54f6d" containerName="discovery" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479087 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11439f87-11d4-4015-853c-b5d587d54f6d" containerName="discovery" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479101 4711 cpu_manager.go:414] "containerMap: removing container" podUID="cc40b733-4aaf-445c-a516-53810817296c" containerName="maas-api" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479110 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc40b733-4aaf-445c-a516-53810817296c" containerName="maas-api" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479129 4711 cpu_manager.go:414] "containerMap: removing container" podUID="c281bd1d-9536-4754-8f2c-75596c7f9ac6" containerName="maas-api" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479137 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c281bd1d-9536-4754-8f2c-75596c7f9ac6" containerName="maas-api" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479171 4711 cpu_manager.go:414] "containerMap: removing container" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479179 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479203 4711 cpu_manager.go:414] "containerMap: removing container" podUID="0f5b00b9-b90b-47de-8345-4937e5bb688c" containerName="maas-api" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479210 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5b00b9-b90b-47de-8345-4937e5bb688c" containerName="maas-api" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479219 4711 cpu_manager.go:414] "containerMap: removing container" podUID="cc0e4e8a-eef2-428c-8b18-2507a7dc7a64" containerName="payload-processing" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479226 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0e4e8a-eef2-428c-8b18-2507a7dc7a64" containerName="payload-processing" Nov 03 20:57:27.479264 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479271 4711 cpu_manager.go:414] "containerMap: removing container" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479279 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479467 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3f57ad-c030-4739-8723-f728de7a129a" containerName="istio-proxy" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479489 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="85954b98-68ed-4e4f-8ba2-bd7c6dcf28c9" containerName="istio-proxy" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479499 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="11439f87-11d4-4015-853c-b5d587d54f6d" containerName="discovery" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479510 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc40b733-4aaf-445c-a516-53810817296c" containerName="maas-api" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479521 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="c281bd1d-9536-4754-8f2c-75596c7f9ac6" containerName="maas-api" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479535 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479545 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5b00b9-b90b-47de-8345-4937e5bb688c" containerName="maas-api" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479553 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0e4e8a-eef2-428c-8b18-2507a7dc7a64" containerName="payload-processing" Nov 03 20:57:27.479754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.479567 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" Nov 03 20:57:27.485135 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.485118 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.493323 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.493293 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j"] Nov 03 20:57:27.521664 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.521634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:27.548598 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.548570 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-63fde4-openshift-default-dockercfg-7nd6f\"" Nov 03 20:57:27.599220 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.599191 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-socket\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.599365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.599239 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-data\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.599365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.599298 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmsq\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-kube-api-access-fgmsq\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.599365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.599319 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-podinfo\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.599561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.599509 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-envoy\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.599634 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.599573 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-token\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.599682 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.599638 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-credential-socket\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.599717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.599684 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-certs\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.599717 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.599707 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istiod-ca-cert\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.706511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.706466 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-credential-socket\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.735659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.735577 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-certs\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.736530 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.736503 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istiod-ca-cert\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.737652 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.737631 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-socket\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.739857 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.739837 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-data\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.740707 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.740685 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmsq\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-kube-api-access-fgmsq\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.754322 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.754301 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-podinfo\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.756100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.754605 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-envoy\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.756100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.754705 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-token\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.756100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.708255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:27.756100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.722368 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-credential-socket\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.756100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.737276 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istiod-ca-cert\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.756100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.739510 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-socket\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.756100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.740411 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-data\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.756100 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.736394 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-certs\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.779128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.779096 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-envoy\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.783368 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.783312 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-podinfo\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.803050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.802981 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-token\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.822813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.808078 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmsq\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-kube-api-access-fgmsq\") pod \"e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:27.890994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:27.890953 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:28.102382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:28.102344 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:28.296692 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:28.296655 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:28.399515 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:28.399481 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j"] Nov 03 20:57:28.399803 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:57:28.399781 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58b4c01_2f45_4b9b_a76c_c6d31c100337.slice/crio-cb6a201a066f3902a9c65f814a77601e72e28eae3847f9610e2bd08c63a3a8a9 WatchSource:0}: Error finding container cb6a201a066f3902a9c65f814a77601e72e28eae3847f9610e2bd08c63a3a8a9: Status 404 returned error can't find the container with id cb6a201a066f3902a9c65f814a77601e72e28eae3847f9610e2bd08c63a3a8a9 Nov 03 20:57:28.403542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:28.403502 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:57:28.403650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:28.403598 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:57:28.403650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:28.403640 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:57:28.419023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:28.418989 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" event={"ID":"f58b4c01-2f45-4b9b-a76c-c6d31c100337","Type":"ContainerStarted","Data":"cb6a201a066f3902a9c65f814a77601e72e28eae3847f9610e2bd08c63a3a8a9"} Nov 03 20:57:29.117668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:29.117602 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:29.427209 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:29.427118 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" event={"ID":"f58b4c01-2f45-4b9b-a76c-c6d31c100337","Type":"ContainerStarted","Data":"8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd"} Nov 03 20:57:29.452761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:29.452706 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" podStartSLOduration=2.452693889 podStartE2EDuration="2.452693889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:29.451698863 +0000 UTC m=+3352.901804929" watchObservedRunningTime="2026-11-03 20:57:29.452693889 +0000 UTC m=+3352.902799934" Nov 03 20:57:29.855907 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:29.855871 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:30.103008 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:30.102979 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:30.109390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:30.109324 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:30.435578 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:30.435493 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:30.436396 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:30.436379 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:57:30.886620 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:30.886586 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:31.591387 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:31.591351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:31.762491 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:31.752523 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:32.596360 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.596268 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:32.842260 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.842226 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:32.844520 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.844494 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd"] Nov 03 20:57:32.845210 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.845195 4711 cpu_manager.go:414] "containerMap: removing container" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" Nov 03 20:57:32.845254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.845212 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77bf29b7-c7b6-4200-b2f3-0bdbe95e29d9" containerName="istio-proxy" Nov 03 20:57:32.849734 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.849694 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:32.855761 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.855732 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-63fde4\"" Nov 03 20:57:32.858844 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.858816 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd"] Nov 03 20:57:32.883676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.883648 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t"] Nov 03 20:57:32.892527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.892299 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:32.900063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.900032 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-63fde4-dockercfg-9rd58\"" Nov 03 20:57:32.901838 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.901806 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t"] Nov 03 20:57:32.942080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.942048 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be03ba9-8f44-4bac-a84d-e91310aca456-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t\" (UID: \"9be03ba9-8f44-4bac-a84d-e91310aca456\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:32.942272 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.942113 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13f2e6b-98a3-4200-be21-b6b28db4d38e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd\" (UID: \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:32.942357 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.942329 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp92z\" (UniqueName: \"kubernetes.io/projected/9be03ba9-8f44-4bac-a84d-e91310aca456-kube-api-access-cp92z\") pod \"payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t\" (UID: \"9be03ba9-8f44-4bac-a84d-e91310aca456\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:32.942537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:32.942522 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kprcd\" (UniqueName: \"kubernetes.io/projected/c13f2e6b-98a3-4200-be21-b6b28db4d38e-kube-api-access-kprcd\") pod \"payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd\" (UID: \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:33.042957 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.042921 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:33.044141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.044107 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kprcd\" (UniqueName: \"kubernetes.io/projected/c13f2e6b-98a3-4200-be21-b6b28db4d38e-kube-api-access-kprcd\") pod \"payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd\" (UID: \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:33.044393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.044375 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be03ba9-8f44-4bac-a84d-e91310aca456-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t\" (UID: \"9be03ba9-8f44-4bac-a84d-e91310aca456\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:33.044563 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.044546 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13f2e6b-98a3-4200-be21-b6b28db4d38e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd\" (UID: \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:33.044784 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.044769 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp92z\" (UniqueName: \"kubernetes.io/projected/9be03ba9-8f44-4bac-a84d-e91310aca456-kube-api-access-cp92z\") pod \"payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t\" (UID: \"9be03ba9-8f44-4bac-a84d-e91310aca456\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:33.056472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.046126 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be03ba9-8f44-4bac-a84d-e91310aca456-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t\" (UID: \"9be03ba9-8f44-4bac-a84d-e91310aca456\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:33.056668 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.055225 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13f2e6b-98a3-4200-be21-b6b28db4d38e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd\" (UID: \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:33.079057 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.078995 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp92z\" (UniqueName: \"kubernetes.io/projected/9be03ba9-8f44-4bac-a84d-e91310aca456-kube-api-access-cp92z\") pod \"payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t\" (UID: \"9be03ba9-8f44-4bac-a84d-e91310aca456\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:33.120473 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.120384 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kprcd\" (UniqueName: \"kubernetes.io/projected/c13f2e6b-98a3-4200-be21-b6b28db4d38e-kube-api-access-kprcd\") pod \"payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd\" (UID: \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:33.170840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.170799 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:33.225464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.224717 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:33.524328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.524302 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t"] Nov 03 20:57:33.525264 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:57:33.525238 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be03ba9_8f44_4bac_a84d_e91310aca456.slice/crio-5af753752ae42f7b339b02e5cb78fcecfadddb26d40575286155638236d0493a WatchSource:0}: Error finding container 5af753752ae42f7b339b02e5cb78fcecfadddb26d40575286155638236d0493a: Status 404 returned error can't find the container with id 5af753752ae42f7b339b02e5cb78fcecfadddb26d40575286155638236d0493a Nov 03 20:57:33.532526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.532501 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd"] Nov 03 20:57:33.532899 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:57:33.532874 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13f2e6b_98a3_4200_be21_b6b28db4d38e.slice/crio-f732dca2ac44e45ba572fc274b0ce72bedaf4f548042fff0d544fe68380751af WatchSource:0}: Error finding container f732dca2ac44e45ba572fc274b0ce72bedaf4f548042fff0d544fe68380751af: Status 404 returned error can't find the container with id f732dca2ac44e45ba572fc274b0ce72bedaf4f548042fff0d544fe68380751af Nov 03 20:57:33.709379 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:33.709346 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:34.361789 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.361749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:34.403630 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.403489 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll"] Nov 03 20:57:34.403809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.403768 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" containerName="istio-proxy" containerID="cri-o://cffbf7c5077f157e401815a06e69090d4a133b08e90ba6e898afc1e12ba67470" gracePeriod=30 Nov 03 20:57:34.464168 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.464130 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" event={"ID":"9be03ba9-8f44-4bac-a84d-e91310aca456","Type":"ContainerStarted","Data":"cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942"} Nov 03 20:57:34.464381 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.464195 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" event={"ID":"9be03ba9-8f44-4bac-a84d-e91310aca456","Type":"ContainerStarted","Data":"5af753752ae42f7b339b02e5cb78fcecfadddb26d40575286155638236d0493a"} Nov 03 20:57:34.464381 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.464315 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:34.465713 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.465691 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" event={"ID":"c13f2e6b-98a3-4200-be21-b6b28db4d38e","Type":"ContainerStarted","Data":"c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed"} Nov 03 20:57:34.465828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.465725 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" event={"ID":"c13f2e6b-98a3-4200-be21-b6b28db4d38e","Type":"ContainerStarted","Data":"f732dca2ac44e45ba572fc274b0ce72bedaf4f548042fff0d544fe68380751af"} Nov 03 20:57:34.465828 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.465814 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:34.501255 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.501185 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" podStartSLOduration=2.5011629319999997 podStartE2EDuration="2.501162932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:34.495842476 +0000 UTC m=+3357.945948522" watchObservedRunningTime="2026-11-03 20:57:34.501162932 +0000 UTC m=+3357.951268979" Nov 03 20:57:34.528680 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:34.528628 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" podStartSLOduration=2.528613482 podStartE2EDuration="2.528613482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:34.525695575 +0000 UTC m=+3357.975801620" watchObservedRunningTime="2026-11-03 20:57:34.528613482 +0000 UTC m=+3357.978719528" Nov 03 20:57:35.199795 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:35.199758 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:35.720935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:35.720904 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:35.853352 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:35.853317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:35.955137 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:35.955106 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9"] Nov 03 20:57:35.955354 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:35.955331 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" containerID="cri-o://ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4" gracePeriod=30 Nov 03 20:57:35.991832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:35.991739 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:36.702553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:36.702513 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:37.353686 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:37.353652 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:38.225539 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:38.225292 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:38.911809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:38.911769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:39.588889 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.587490 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:39.727524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.727501 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:57:39.785455 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785419 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-credential-socket\") pod \"1c6aa508-a110-46f4-b823-c4b46e04bb55\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " Nov 03 20:57:39.785637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785491 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6aa508-a110-46f4-b823-c4b46e04bb55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6aa508-a110-46f4-b823-c4b46e04bb55-istiod-ca-cert\") pod \"1c6aa508-a110-46f4-b823-c4b46e04bb55\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " Nov 03 20:57:39.785637 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785616 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-socket\") pod \"1c6aa508-a110-46f4-b823-c4b46e04bb55\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " Nov 03 20:57:39.785772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785668 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-envoy\") pod \"1c6aa508-a110-46f4-b823-c4b46e04bb55\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " Nov 03 20:57:39.785772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785701 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-credential-socket" pod "1c6aa508-a110-46f4-b823-c4b46e04bb55" (UID: "1c6aa508-a110-46f4-b823-c4b46e04bb55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:39.785772 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785744 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-certs\") pod \"1c6aa508-a110-46f4-b823-c4b46e04bb55\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " Nov 03 20:57:39.785953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785781 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-token\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-token\") pod \"1c6aa508-a110-46f4-b823-c4b46e04bb55\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " Nov 03 20:57:39.785953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785840 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-kube-api-access-c87pq\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-kube-api-access-c87pq\") pod \"1c6aa508-a110-46f4-b823-c4b46e04bb55\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " Nov 03 20:57:39.785953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785918 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-socket" pod "1c6aa508-a110-46f4-b823-c4b46e04bb55" (UID: "1c6aa508-a110-46f4-b823-c4b46e04bb55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:39.785953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785933 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6aa508-a110-46f4-b823-c4b46e04bb55-istiod-ca-cert" pod "1c6aa508-a110-46f4-b823-c4b46e04bb55" (UID: "1c6aa508-a110-46f4-b823-c4b46e04bb55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:39.785953 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.785943 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-data\") pod \"1c6aa508-a110-46f4-b823-c4b46e04bb55\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " Nov 03 20:57:39.786218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.786034 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-podinfo\") pod \"1c6aa508-a110-46f4-b823-c4b46e04bb55\" (UID: \"1c6aa508-a110-46f4-b823-c4b46e04bb55\") " Nov 03 20:57:39.786218 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.786202 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-certs" pod "1c6aa508-a110-46f4-b823-c4b46e04bb55" (UID: "1c6aa508-a110-46f4-b823-c4b46e04bb55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:39.786496 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.786458 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-data" pod "1c6aa508-a110-46f4-b823-c4b46e04bb55" (UID: "1c6aa508-a110-46f4-b823-c4b46e04bb55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:39.786763 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.786747 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.786875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.786766 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.786875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.786781 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.786875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.786796 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c6aa508-a110-46f4-b823-c4b46e04bb55-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.786875 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.786810 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.788855 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.788820 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-envoy" pod "1c6aa508-a110-46f4-b823-c4b46e04bb55" (UID: "1c6aa508-a110-46f4-b823-c4b46e04bb55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:39.789261 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.789232 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-podinfo" pod "1c6aa508-a110-46f4-b823-c4b46e04bb55" (UID: "1c6aa508-a110-46f4-b823-c4b46e04bb55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:39.789390 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.789360 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-token" pod "1c6aa508-a110-46f4-b823-c4b46e04bb55" (UID: "1c6aa508-a110-46f4-b823-c4b46e04bb55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:39.799639 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.799591 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-kube-api-access-c87pq" pod "1c6aa508-a110-46f4-b823-c4b46e04bb55" (UID: "1c6aa508-a110-46f4-b823-c4b46e04bb55"). InnerVolumeSpecName "kube-api-access-c87pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:39.846286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.846262 4711 scope.go:122] "RemoveContainer" containerID="cffbf7c5077f157e401815a06e69090d4a133b08e90ba6e898afc1e12ba67470" Nov 03 20:57:39.888675 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.888646 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.888817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.888681 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.888817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.888698 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87pq\" (UniqueName: \"kubernetes.io/projected/1c6aa508-a110-46f4-b823-c4b46e04bb55-kube-api-access-c87pq\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:39.888817 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:39.888713 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c6aa508-a110-46f4-b823-c4b46e04bb55-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:40.328618 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.328579 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:40.455393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.455361 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:57:40.455574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.455421 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:57:40.507127 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.507084 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" event={"ID":"1c6aa508-a110-46f4-b823-c4b46e04bb55","Type":"ContainerDied","Data":"cffbf7c5077f157e401815a06e69090d4a133b08e90ba6e898afc1e12ba67470"} Nov 03 20:57:40.507284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.507156 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" event={"ID":"1c6aa508-a110-46f4-b823-c4b46e04bb55","Type":"ContainerDied","Data":"29c90b6d19bcfea63bd077b0877147c43b71e8290f28609362d434818f528313"} Nov 03 20:57:40.507284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.507195 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll" Nov 03 20:57:40.550425 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.550398 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll"] Nov 03 20:57:40.554503 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.554478 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-05473b-openshift-default-5d6455945-b9tll"] Nov 03 20:57:40.682397 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.682320 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" path="/var/lib/kubelet/pods/1c6aa508-a110-46f4-b823-c4b46e04bb55/volumes" Nov 03 20:57:40.998537 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:40.998492 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:41.235402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.235349 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:57:41.308429 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.308375 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-credential-socket\") pod \"78f223f7-bc76-4228-b969-6c11f753219e\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " Nov 03 20:57:41.308666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.308534 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-data\") pod \"78f223f7-bc76-4228-b969-6c11f753219e\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " Nov 03 20:57:41.308666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.308570 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-certs\") pod \"78f223f7-bc76-4228-b969-6c11f753219e\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " Nov 03 20:57:41.308666 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.308628 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78f223f7-bc76-4228-b969-6c11f753219e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f223f7-bc76-4228-b969-6c11f753219e-istio-podinfo\") pod \"78f223f7-bc76-4228-b969-6c11f753219e\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " Nov 03 20:57:41.308939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.308680 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-envoy\") pod \"78f223f7-bc76-4228-b969-6c11f753219e\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " Nov 03 20:57:41.308939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.308737 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-istio-token\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-istio-token\") pod \"78f223f7-bc76-4228-b969-6c11f753219e\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " Nov 03 20:57:41.308939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.308793 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78f223f7-bc76-4228-b969-6c11f753219e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f223f7-bc76-4228-b969-6c11f753219e-istiod-ca-cert\") pod \"78f223f7-bc76-4228-b969-6c11f753219e\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " Nov 03 20:57:41.308939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.308829 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-socket\") pod \"78f223f7-bc76-4228-b969-6c11f753219e\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " Nov 03 20:57:41.308939 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.308897 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-kube-api-access-rkzlx\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-kube-api-access-rkzlx\") pod \"78f223f7-bc76-4228-b969-6c11f753219e\" (UID: \"78f223f7-bc76-4228-b969-6c11f753219e\") " Nov 03 20:57:41.309899 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.309848 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-credential-socket" pod "78f223f7-bc76-4228-b969-6c11f753219e" (UID: "78f223f7-bc76-4228-b969-6c11f753219e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:41.310233 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.310212 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-socket" pod "78f223f7-bc76-4228-b969-6c11f753219e" (UID: "78f223f7-bc76-4228-b969-6c11f753219e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:41.310329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.310235 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f223f7-bc76-4228-b969-6c11f753219e-istiod-ca-cert" pod "78f223f7-bc76-4228-b969-6c11f753219e" (UID: "78f223f7-bc76-4228-b969-6c11f753219e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:41.310405 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.310387 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-certs" pod "78f223f7-bc76-4228-b969-6c11f753219e" (UID: "78f223f7-bc76-4228-b969-6c11f753219e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:41.310574 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.310549 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-data" pod "78f223f7-bc76-4228-b969-6c11f753219e" (UID: "78f223f7-bc76-4228-b969-6c11f753219e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:41.312819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.312688 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-kube-api-access-rkzlx" pod "78f223f7-bc76-4228-b969-6c11f753219e" (UID: "78f223f7-bc76-4228-b969-6c11f753219e"). InnerVolumeSpecName "kube-api-access-rkzlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:41.312914 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.312819 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78f223f7-bc76-4228-b969-6c11f753219e-istio-podinfo" pod "78f223f7-bc76-4228-b969-6c11f753219e" (UID: "78f223f7-bc76-4228-b969-6c11f753219e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:41.313061 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.313033 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-istio-token" pod "78f223f7-bc76-4228-b969-6c11f753219e" (UID: "78f223f7-bc76-4228-b969-6c11f753219e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:41.313236 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.313218 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-envoy" pod "78f223f7-bc76-4228-b969-6c11f753219e" (UID: "78f223f7-bc76-4228-b969-6c11f753219e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:41.411275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.411241 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.411275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.411270 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkzlx\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-kube-api-access-rkzlx\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.411275 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.411281 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.411531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.411289 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.411531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.411298 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.411531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.411306 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78f223f7-bc76-4228-b969-6c11f753219e-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.411531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.411315 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78f223f7-bc76-4228-b969-6c11f753219e-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.411531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.411325 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78f223f7-bc76-4228-b969-6c11f753219e-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.411531 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.411333 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78f223f7-bc76-4228-b969-6c11f753219e-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:41.515026 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.514994 4711 generic.go:356] "Generic (PLEG): container finished" podID="78f223f7-bc76-4228-b969-6c11f753219e" containerID="ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4" exitCode=0 Nov 03 20:57:41.515191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.515049 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" Nov 03 20:57:41.515191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.515073 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" event={"ID":"78f223f7-bc76-4228-b969-6c11f753219e","Type":"ContainerDied","Data":"ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4"} Nov 03 20:57:41.515191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.515152 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9" event={"ID":"78f223f7-bc76-4228-b969-6c11f753219e","Type":"ContainerDied","Data":"9b73ff749c2f77ee65bc889d7bb0d04bde84b36d9510ab0c69d66b0319ad4bcc"} Nov 03 20:57:41.515191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.515180 4711 scope.go:122] "RemoveContainer" containerID="ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4" Nov 03 20:57:41.529677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.529657 4711 scope.go:122] "RemoveContainer" containerID="ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4" Nov 03 20:57:41.530227 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:57:41.530198 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4\": container with ID starting with ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4 not found: ID does not exist" containerID="ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4" Nov 03 20:57:41.530328 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.530230 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4"} err="failed to get container status \"ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4\": rpc error: code = NotFound desc = could not find container \"ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4\": container with ID starting with ca50844dfecd3ba0ea88c1e5e615a454a2300f401a0e24951f2da2837ae9a6b4 not found: ID does not exist" Nov 03 20:57:41.543029 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.543007 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9"] Nov 03 20:57:41.550807 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.550787 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bab2c3-openshift-default-748796c95-f58z9"] Nov 03 20:57:41.660798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:41.660766 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:42.404358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:42.404319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:42.691253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:42.691169 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78f223f7-bc76-4228-b969-6c11f753219e" path="/var/lib/kubelet/pods/78f223f7-bc76-4228-b969-6c11f753219e/volumes" Nov 03 20:57:43.157470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:43.149409 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:43.295927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:43.295892 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:43.990627 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:43.990570 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:44.690126 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:44.690091 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:45.475526 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:45.475494 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 20:57:45.475903 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:45.475559 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 20:57:45.525677 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:45.525641 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:46.306591 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:46.306548 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:46.991410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:46.991376 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:47.532258 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:57:47.532222 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:57:47.721191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:47.721149 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:48.255002 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:48.254963 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:48.396694 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:48.396656 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:48.543073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:48.542982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:48.677863 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:48.677813 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:49.245470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:49.228094 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:50.046086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.046049 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:50.730658 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.730623 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk"] Nov 03 20:57:50.731371 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.731350 4711 cpu_manager.go:414] "containerMap: removing container" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" containerName="istio-proxy" Nov 03 20:57:50.731718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.731373 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" containerName="istio-proxy" Nov 03 20:57:50.731718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.731493 4711 cpu_manager.go:414] "containerMap: removing container" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" Nov 03 20:57:50.731718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.731507 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" Nov 03 20:57:50.731718 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.731711 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f223f7-bc76-4228-b969-6c11f753219e" containerName="istio-proxy" Nov 03 20:57:50.731975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.731732 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6aa508-a110-46f4-b823-c4b46e04bb55" containerName="istio-proxy" Nov 03 20:57:50.737030 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.737003 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:50.757548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.757470 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk"] Nov 03 20:57:50.770068 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.770034 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-63fde4-serving-cert\"" Nov 03 20:57:50.823740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.823705 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:50.892248 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.892201 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:50.892509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.892339 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:50.892509 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.892394 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4dx\" (UniqueName: \"kubernetes.io/projected/739a3255-da88-41b3-bd81-331cdce0a6be-kube-api-access-vx4dx\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:50.892676 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.892657 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/739a3255-da88-41b3-bd81-331cdce0a6be-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:50.971885 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.970043 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:50.995169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.995022 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/739a3255-da88-41b3-bd81-331cdce0a6be-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:50.995358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.995245 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:50.995358 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.995313 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:50.995504 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.995393 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4dx\" (UniqueName: \"kubernetes.io/projected/739a3255-da88-41b3-bd81-331cdce0a6be-kube-api-access-vx4dx\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:50.996549 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:50.996484 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/739a3255-da88-41b3-bd81-331cdce0a6be-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:51.005740 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:51.005708 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:51.008529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:51.008423 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4dx\" (UniqueName: \"kubernetes.io/projected/739a3255-da88-41b3-bd81-331cdce0a6be-kube-api-access-vx4dx\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:51.009291 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:51.009267 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:51.061621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:51.061583 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:51.390317 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:51.390285 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk"] Nov 03 20:57:51.394416 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:57:51.394323 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739a3255_da88_41b3_bd81_331cdce0a6be.slice/crio-7fbd7867f6d2067908a43f79fa6314ec731522732a826c7c28d65a23dc2c027f WatchSource:0}: Error finding container 7fbd7867f6d2067908a43f79fa6314ec731522732a826c7c28d65a23dc2c027f: Status 404 returned error can't find the container with id 7fbd7867f6d2067908a43f79fa6314ec731522732a826c7c28d65a23dc2c027f Nov 03 20:57:51.396089 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:51.396058 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:57:51.590286 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:51.590253 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" event={"ID":"739a3255-da88-41b3-bd81-331cdce0a6be","Type":"ContainerStarted","Data":"7fbd7867f6d2067908a43f79fa6314ec731522732a826c7c28d65a23dc2c027f"} Nov 03 20:57:51.702069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:51.701950 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:52.575901 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:52.575859 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:52.641660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:52.641617 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" event={"ID":"739a3255-da88-41b3-bd81-331cdce0a6be","Type":"ContainerStarted","Data":"b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d"} Nov 03 20:57:52.641876 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:52.641720 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:52.662439 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:52.662370 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" podStartSLOduration=2.1773439 podStartE2EDuration="2.66235194s" totalImagesPullingTime="485.00804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:52.661508755 +0000 UTC m=+3376.111614803" watchObservedRunningTime="2026-11-03 20:57:52.66235194 +0000 UTC m=+3376.112457987" Nov 03 20:57:53.394809 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:53.394769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:54.204472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:54.204393 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:54.723151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:54.723100 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:55.153343 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:55.153279 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:55.744365 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:55.744316 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:55.877315 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:55.877276 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:56.253309 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:56.253263 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:56.797745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:56.797712 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:57.569684 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:57.569646 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:58.449470 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:58.449260 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:58.654465 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:58.654427 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 20:57:59.344120 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:59.344015 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:57:59.596094 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:57:59.595998 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:00.096541 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:00.096508 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:00.832412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:00.832373 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:00.973635 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:00.973598 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:01.318783 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:01.318750 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:01.471224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:01.471155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:01.955660 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:01.955620 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:02.690614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:02.690580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:02.819743 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:02.819696 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:03.510342 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:03.510307 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:04.271615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:04.271581 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:04.410719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:04.410687 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:05.322741 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:05.322691 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:06.292671 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:06.292634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:07.183779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:07.183743 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:08.060086 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:08.060046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:08.717229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:08.717193 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:09.015254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:09.015205 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:09.535498 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:09.535435 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:10.362577 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:10.362541 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:10.508617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:10.508583 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:10.863731 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:10.863698 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:11.616806 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:11.616771 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:12.381020 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:12.380940 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:13.250481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:13.250030 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:14.033259 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:14.033212 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:14.845695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:14.845652 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:15.618362 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:15.618320 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:16.396779 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:16.396724 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:16.880205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:16.880171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:17.012252 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:17.012211 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:17.348394 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:17.348360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:17.546329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:17.546238 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:17.582117 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:58:17.582080 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:58:17.905014 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:17.904979 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:18.308529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.308490 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx"] Nov 03 20:58:18.323095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.321009 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.323095 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.322494 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx"] Nov 03 20:58:18.342202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.342125 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:18.349311 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.349283 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtq5z\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-kube-api-access-xtq5z\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.349650 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.349616 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-certs\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.349861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.349843 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-token\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.350007 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.349992 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-credential-socket\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.350188 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.350159 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-podinfo\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.350337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.350320 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-data\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.350398 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.350363 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-envoy\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.350499 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.350482 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159c4fe1-8080-4f38-b1d9-d67250329c9a-istiod-ca-cert\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.350561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.350521 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-socket\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.380246 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.380208 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3bcc2b-openshift-default-dockercfg-kc6vh\"" Nov 03 20:58:18.451773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.451738 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-data\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.451773 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.451783 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-envoy\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.451853 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159c4fe1-8080-4f38-b1d9-d67250329c9a-istiod-ca-cert\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.451877 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-socket\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.451918 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtq5z\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-kube-api-access-xtq5z\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.451959 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-certs\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452017 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.452014 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-token\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.452067 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-credential-socket\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.452148 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-podinfo\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.452532 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-socket\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.452586 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-credential-socket\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.452910 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.452619 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159c4fe1-8080-4f38-b1d9-d67250329c9a-istiod-ca-cert\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.453016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.452772 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-certs\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.453016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.452793 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-data\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.454553 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.454523 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-envoy\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.454629 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.454588 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-podinfo\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.464205 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.464177 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-token\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.466440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.466413 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtq5z\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-kube-api-access-xtq5z\") pod \"e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.479069 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.479029 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:18.617519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.617335 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:18.646533 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.645759 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:18.979478 ip-10-0-142-5 kubenswrapper[4711]: W1103 20:58:18.977807 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159c4fe1_8080_4f38_b1d9_d67250329c9a.slice/crio-b3566f42d236981b1ec14a6aaa9b4cfeadaa54aa5e30f44b0aa454a95a00d6bc WatchSource:0}: Error finding container b3566f42d236981b1ec14a6aaa9b4cfeadaa54aa5e30f44b0aa454a95a00d6bc: Status 404 returned error can't find the container with id b3566f42d236981b1ec14a6aaa9b4cfeadaa54aa5e30f44b0aa454a95a00d6bc Nov 03 20:58:18.979478 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.979222 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx"] Nov 03 20:58:18.981599 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.981557 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:58:18.981745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.981641 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:58:18.981745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:18.981683 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:58:19.305845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:19.305808 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:19.937873 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:19.937836 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" event={"ID":"159c4fe1-8080-4f38-b1d9-d67250329c9a","Type":"ContainerStarted","Data":"8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02"} Nov 03 20:58:19.938349 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:19.937905 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" event={"ID":"159c4fe1-8080-4f38-b1d9-d67250329c9a","Type":"ContainerStarted","Data":"b3566f42d236981b1ec14a6aaa9b4cfeadaa54aa5e30f44b0aa454a95a00d6bc"} Nov 03 20:58:19.965240 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:19.965184 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" podStartSLOduration=1.96516672 podStartE2EDuration="1.96516672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:19.96413544 +0000 UTC m=+3403.414241498" watchObservedRunningTime="2026-11-03 20:58:19.96516672 +0000 UTC m=+3403.415272769" Nov 03 20:58:19.970704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:19.970664 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:20.235532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:20.231025 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:20.647097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:20.647060 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:20.653511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:20.653485 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:20.944329 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:20.944245 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:20.945213 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:20.945193 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:58:21.088466 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:21.087251 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:21.952719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:21.952685 4711 generic.go:356] "Generic (PLEG): container finished" podID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerID="8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd" exitCode=0 Nov 03 20:58:21.953111 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:21.952765 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" event={"ID":"f58b4c01-2f45-4b9b-a76c-c6d31c100337","Type":"ContainerDied","Data":"8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd"} Nov 03 20:58:21.953229 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:21.953211 4711 scope.go:122] "RemoveContainer" containerID="8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd" Nov 03 20:58:21.954938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:21.954911 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:58:21.955105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:21.954979 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:58:21.955105 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:21.955013 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:58:22.018183 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:22.018134 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:22.161615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:22.161534 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:22.423513 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:22.423480 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:22.961819 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:22.961783 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" event={"ID":"f58b4c01-2f45-4b9b-a76c-c6d31c100337","Type":"ContainerStarted","Data":"bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc"} Nov 03 20:58:23.102547 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:23.102509 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:58:23.102724 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:23.102695 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:58:23.102777 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:23.102754 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 20:58:23.267464 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:23.264711 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:23.592698 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:23.592603 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:23.845845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:23.845750 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:24.102719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:24.102620 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:58:24.102719 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:24.102675 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 20:58:24.212832 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:24.212791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:24.991474 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:24.991320 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:25.103038 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:25.102997 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:58:25.103529 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:25.103053 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 20:58:25.669472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:25.666248 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:25.832467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:25.829993 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:26.062866 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:26.062830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:26.104018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:26.103982 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:58:26.104785 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:26.104655 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 20:58:26.199608 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:26.199570 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:26.603840 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:26.603806 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:26.737422 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:26.737242 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:27.039298 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:27.039217 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:27.105467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:27.105148 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:58:27.105467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:27.105225 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 20:58:27.394845 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:27.394558 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:27.805075 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:27.805042 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:27.937745 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:27.937679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:28.102798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:28.102709 4711 patch_prober.go:29] interesting pod/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Nov 03 20:58:28.102798 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:28.102776 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Nov 03 20:58:28.536377 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:28.536338 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:29.107882 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:29.107853 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:58:29.108292 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:29.108273 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:58:29.109040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:29.109021 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 20:58:29.305562 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:29.305528 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:30.020307 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:30.020263 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:30.699063 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:30.699031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:31.323554 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:31.323510 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:32.021252 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:32.021186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:32.160253 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:32.160220 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:32.293617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:32.293516 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:32.653099 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:32.653002 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:32.988294 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:32.988208 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:33.178382 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:33.178348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:33.818874 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:33.818829 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:34.019558 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:34.019517 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:34.847532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:34.844303 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:35.561500 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:35.561458 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:35.730407 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:35.730355 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:36.388400 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:36.388367 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:37.223191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:37.223150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:37.911994 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:37.911954 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:38.598872 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:38.598826 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:39.282493 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:39.282407 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:40.051166 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:40.051136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:40.182988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:40.182896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:40.891841 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:40.891794 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:41.099489 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:41.099429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:41.718712 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:41.718675 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:41.855868 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:41.855770 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:42.274373 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:42.274327 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:42.500050 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:42.500015 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:43.307393 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:43.307357 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:44.009934 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:44.009894 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:44.665519 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:44.665473 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:45.436704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:45.436653 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:46.132629 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:46.132585 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:46.660720 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:46.660679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:46.998207 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:46.998118 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:47.131888 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:47.131854 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:47.532356 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:58:47.532321 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:58:47.711047 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:47.711013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:48.299018 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:48.298979 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:48.437327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:48.437291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:48.583121 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:48.583026 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:49.010128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:49.010097 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:49.270561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:49.259377 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp"] Nov 03 20:58:49.270561 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:49.259622 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" containerID="cri-o://094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be" gracePeriod=30 Nov 03 20:58:49.312870 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:49.312815 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:49.719092 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:49.718995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:50.129271 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.129213 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:50.458438 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.458356 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt"] Nov 03 20:58:50.459224 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.459187 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" podUID="437f7610-bc66-4f26-9a3f-b751fb523aaf" containerName="maas-api" containerID="cri-o://f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1" gracePeriod=30 Nov 03 20:58:50.536647 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.536615 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:50.880404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.880375 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:58:50.988481 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.988439 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-maas-api-tls\") pod \"437f7610-bc66-4f26-9a3f-b751fb523aaf\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " Nov 03 20:58:50.988659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.988558 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-metrics-certs\") pod \"437f7610-bc66-4f26-9a3f-b751fb523aaf\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " Nov 03 20:58:50.988659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.988605 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/437f7610-bc66-4f26-9a3f-b751fb523aaf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/437f7610-bc66-4f26-9a3f-b751fb523aaf-openshift-service-ca\") pod \"437f7610-bc66-4f26-9a3f-b751fb523aaf\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " Nov 03 20:58:50.988846 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.988825 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437f7610-bc66-4f26-9a3f-b751fb523aaf-kube-api-access-6tmjb\" (UniqueName: \"kubernetes.io/projected/437f7610-bc66-4f26-9a3f-b751fb523aaf-kube-api-access-6tmjb\") pod \"437f7610-bc66-4f26-9a3f-b751fb523aaf\" (UID: \"437f7610-bc66-4f26-9a3f-b751fb523aaf\") " Nov 03 20:58:50.989022 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.989001 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437f7610-bc66-4f26-9a3f-b751fb523aaf-openshift-service-ca" pod "437f7610-bc66-4f26-9a3f-b751fb523aaf" (UID: "437f7610-bc66-4f26-9a3f-b751fb523aaf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:50.989413 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.989390 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/437f7610-bc66-4f26-9a3f-b751fb523aaf-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:50.990644 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.990620 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-maas-api-tls" pod "437f7610-bc66-4f26-9a3f-b751fb523aaf" (UID: "437f7610-bc66-4f26-9a3f-b751fb523aaf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:50.990815 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.990789 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-metrics-certs" pod "437f7610-bc66-4f26-9a3f-b751fb523aaf" (UID: "437f7610-bc66-4f26-9a3f-b751fb523aaf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:58:50.990938 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:50.990904 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437f7610-bc66-4f26-9a3f-b751fb523aaf-kube-api-access-6tmjb" pod "437f7610-bc66-4f26-9a3f-b751fb523aaf" (UID: "437f7610-bc66-4f26-9a3f-b751fb523aaf"). InnerVolumeSpecName "kube-api-access-6tmjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:51.001693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.001667 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:51.091337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.091245 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmjb\" (UniqueName: \"kubernetes.io/projected/437f7610-bc66-4f26-9a3f-b751fb523aaf-kube-api-access-6tmjb\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:51.091337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.091274 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:51.091337 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.091284 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/437f7610-bc66-4f26-9a3f-b751fb523aaf-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:51.224919 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.224878 4711 generic.go:356] "Generic (PLEG): container finished" podID="437f7610-bc66-4f26-9a3f-b751fb523aaf" containerID="f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1" exitCode=0 Nov 03 20:58:51.225117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.224926 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" Nov 03 20:58:51.225117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.225060 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" event={"ID":"437f7610-bc66-4f26-9a3f-b751fb523aaf","Type":"ContainerDied","Data":"f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1"} Nov 03 20:58:51.225117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.225098 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt" event={"ID":"437f7610-bc66-4f26-9a3f-b751fb523aaf","Type":"ContainerDied","Data":"518996b3a26da6b5d16edffbb71357b7ceb3cb753594f682013f65a7d64bb8bc"} Nov 03 20:58:51.225252 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.225123 4711 scope.go:122] "RemoveContainer" containerID="f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1" Nov 03 20:58:51.242196 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.242166 4711 scope.go:122] "RemoveContainer" containerID="f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1" Nov 03 20:58:51.242566 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:58:51.242541 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1\": container with ID starting with f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1 not found: ID does not exist" containerID="f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1" Nov 03 20:58:51.242679 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.242580 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1"} err="failed to get container status \"f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1\": rpc error: code = NotFound desc = could not find container \"f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1\": container with ID starting with f85a40592cec6ded559959a52d7a379b1918581b4312d8da08605ac2fc1918a1 not found: ID does not exist" Nov 03 20:58:51.259890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.259861 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt"] Nov 03 20:58:51.263402 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.263381 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5ecee8-bf5c7bb6d-ktnmt"] Nov 03 20:58:51.704614 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:51.704578 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:52.410896 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:52.410858 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:52.453588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:52.453154 4711 patch_prober.go:29] interesting pod/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:52.453588 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:52.453242 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:52.681109 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:52.681020 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="437f7610-bc66-4f26-9a3f-b751fb523aaf" path="/var/lib/kubelet/pods/437f7610-bc66-4f26-9a3f-b751fb523aaf/volumes" Nov 03 20:58:53.075601 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:53.075560 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:53.205659 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:53.205624 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:53.591130 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:53.591092 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:53.941890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:53.941805 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:54.074580 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.074546 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:54.543059 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.542996 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:58:54.662871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.662837 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-data\") pod \"774c317c-a849-4a4f-9921-18004a104b10\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " Nov 03 20:58:54.663280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.662906 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-socket\") pod \"774c317c-a849-4a4f-9921-18004a104b10\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " Nov 03 20:58:54.663280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.662957 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/774c317c-a849-4a4f-9921-18004a104b10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774c317c-a849-4a4f-9921-18004a104b10-istio-podinfo\") pod \"774c317c-a849-4a4f-9921-18004a104b10\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " Nov 03 20:58:54.663280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.662993 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-envoy\") pod \"774c317c-a849-4a4f-9921-18004a104b10\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " Nov 03 20:58:54.663280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663072 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-certs\") pod \"774c317c-a849-4a4f-9921-18004a104b10\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " Nov 03 20:58:54.663280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663091 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-data" pod "774c317c-a849-4a4f-9921-18004a104b10" (UID: "774c317c-a849-4a4f-9921-18004a104b10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:54.663280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663102 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-credential-socket\") pod \"774c317c-a849-4a4f-9921-18004a104b10\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " Nov 03 20:58:54.663280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663186 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-istio-token\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-istio-token\") pod \"774c317c-a849-4a4f-9921-18004a104b10\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " Nov 03 20:58:54.663280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663192 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-socket" pod "774c317c-a849-4a4f-9921-18004a104b10" (UID: "774c317c-a849-4a4f-9921-18004a104b10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:54.663280 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663240 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-kube-api-access-68zlj\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-kube-api-access-68zlj\") pod \"774c317c-a849-4a4f-9921-18004a104b10\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " Nov 03 20:58:54.663813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663369 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-certs" pod "774c317c-a849-4a4f-9921-18004a104b10" (UID: "774c317c-a849-4a4f-9921-18004a104b10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:54.663813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663379 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774c317c-a849-4a4f-9921-18004a104b10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774c317c-a849-4a4f-9921-18004a104b10-istiod-ca-cert\") pod \"774c317c-a849-4a4f-9921-18004a104b10\" (UID: \"774c317c-a849-4a4f-9921-18004a104b10\") " Nov 03 20:58:54.663813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663405 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-credential-socket" pod "774c317c-a849-4a4f-9921-18004a104b10" (UID: "774c317c-a849-4a4f-9921-18004a104b10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:54.663813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.663715 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774c317c-a849-4a4f-9921-18004a104b10-istiod-ca-cert" pod "774c317c-a849-4a4f-9921-18004a104b10" (UID: "774c317c-a849-4a4f-9921-18004a104b10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:54.664073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.664055 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774c317c-a849-4a4f-9921-18004a104b10-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:54.664073 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.664073 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:54.664198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.664082 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:54.664198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.664090 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:54.664198 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.664100 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:54.665212 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.665190 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-envoy" pod "774c317c-a849-4a4f-9921-18004a104b10" (UID: "774c317c-a849-4a4f-9921-18004a104b10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:54.665528 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.665511 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-kube-api-access-68zlj" pod "774c317c-a849-4a4f-9921-18004a104b10" (UID: "774c317c-a849-4a4f-9921-18004a104b10"). InnerVolumeSpecName "kube-api-access-68zlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:54.665621 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.665572 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/774c317c-a849-4a4f-9921-18004a104b10-istio-podinfo" pod "774c317c-a849-4a4f-9921-18004a104b10" (UID: "774c317c-a849-4a4f-9921-18004a104b10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:54.665765 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.665751 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-istio-token" pod "774c317c-a849-4a4f-9921-18004a104b10" (UID: "774c317c-a849-4a4f-9921-18004a104b10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:54.765344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.765319 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:54.765344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.765344 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68zlj\" (UniqueName: \"kubernetes.io/projected/774c317c-a849-4a4f-9921-18004a104b10-kube-api-access-68zlj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:54.765559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.765353 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774c317c-a849-4a4f-9921-18004a104b10-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:54.765559 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.765361 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774c317c-a849-4a4f-9921-18004a104b10-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:54.857781 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.857695 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:54.993040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:54.993003 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:55.194342 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.194253 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:55.254216 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.254183 4711 generic.go:356] "Generic (PLEG): container finished" podID="774c317c-a849-4a4f-9921-18004a104b10" containerID="094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be" exitCode=0 Nov 03 20:58:55.254380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.254220 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" event={"ID":"774c317c-a849-4a4f-9921-18004a104b10","Type":"ContainerDied","Data":"094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be"} Nov 03 20:58:55.254380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.254248 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" Nov 03 20:58:55.254380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.254303 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp" event={"ID":"774c317c-a849-4a4f-9921-18004a104b10","Type":"ContainerDied","Data":"6af47a22b4348b65dd4ffc0c4dd3d1f97bc1cc98b6d46aaa8053f44bfa804224"} Nov 03 20:58:55.254380 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.254330 4711 scope.go:122] "RemoveContainer" containerID="094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be" Nov 03 20:58:55.264648 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.264630 4711 scope.go:122] "RemoveContainer" containerID="89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d" Nov 03 20:58:55.273968 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.273946 4711 scope.go:122] "RemoveContainer" containerID="094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be" Nov 03 20:58:55.274255 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:58:55.274230 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be\": container with ID starting with 094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be not found: ID does not exist" containerID="094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be" Nov 03 20:58:55.274346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.274266 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be"} err="failed to get container status \"094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be\": rpc error: code = NotFound desc = could not find container \"094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be\": container with ID starting with 094a0d7fb6c8c997c5930811818f7501acde51c459badfe7aa50297b5f2df6be not found: ID does not exist" Nov 03 20:58:55.274346 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.274289 4711 scope.go:122] "RemoveContainer" containerID="89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d" Nov 03 20:58:55.274568 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:58:55.274549 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d\": container with ID starting with 89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d not found: ID does not exist" containerID="89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d" Nov 03 20:58:55.274626 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.274574 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d"} err="failed to get container status \"89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d\": rpc error: code = NotFound desc = could not find container \"89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d\": container with ID starting with 89e7677a5fe5c76d0cfbeadb4fe09bf8bad8eda3b7249281bcd0be06a245385d not found: ID does not exist" Nov 03 20:58:55.324594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.324560 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp"] Nov 03 20:58:55.348058 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.348028 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-03c633-openshift-default-58bd8ddf77-8wbqp"] Nov 03 20:58:55.903624 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:55.903587 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:56.488534 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:56.488495 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:56.638440 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:56.638398 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:56.690351 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:56.690303 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774c317c-a849-4a4f-9921-18004a104b10" path="/var/lib/kubelet/pods/774c317c-a849-4a4f-9921-18004a104b10/volumes" Nov 03 20:58:56.853639 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:56.853604 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:57.023115 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:57.023083 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:57.728996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:57.728955 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:58.585669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:58.585627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:59.305891 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:59.305666 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:58:59.639490 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:58:59.639396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:00.035113 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:00.035069 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:00.879600 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:00.879563 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:01.011310 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:01.011276 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:01.702927 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:01.702878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:02.513087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:02.513051 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:03.310080 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:03.310045 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:03.448527 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:03.448496 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:03.994070 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:03.994036 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:04.645395 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:04.645360 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:05.401300 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:05.401259 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:06.112949 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:06.112915 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:06.789435 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:06.789404 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:07.503467 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:07.502083 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:07.859153 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:07.859116 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:08.214787 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:08.214691 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:08.890190 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:08.890153 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:09.298988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:09.298953 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:09.950704 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:09.950667 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:10.101119 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:10.101082 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:10.348258 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:10.348217 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:10.481074 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:10.481032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:10.896374 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:10.896339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:11.182842 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:11.182758 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg"] Nov 03 20:59:11.183412 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:11.182999 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" containerID="cri-o://62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2" gracePeriod=30 Nov 03 20:59:11.231254 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:11.231222 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:11.379472 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:11.379269 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:12.051194 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:12.051161 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:12.334287 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:12.334185 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:13.117295 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:13.117184 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:13.796871 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:13.796836 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:14.553191 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:14.553147 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:15.217145 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:15.217109 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:15.904410 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:15.904377 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:16.463021 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.462960 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:59:16.474861 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.474827 4711 generic.go:356] "Generic (PLEG): container finished" podID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerID="62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2" exitCode=0 Nov 03 20:59:16.475028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.474872 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" Nov 03 20:59:16.475028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.474905 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" event={"ID":"d26ab542-92c0-4424-8dd3-d74dc2839bbb","Type":"ContainerDied","Data":"62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2"} Nov 03 20:59:16.475028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.474955 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg" event={"ID":"d26ab542-92c0-4424-8dd3-d74dc2839bbb","Type":"ContainerDied","Data":"d303fc8c7e57c76e7421c4a1d9b5086b2ed2536c248bb9e908bcc257857139cb"} Nov 03 20:59:16.475028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.474980 4711 scope.go:122] "RemoveContainer" containerID="62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2" Nov 03 20:59:16.485284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.485264 4711 scope.go:122] "RemoveContainer" containerID="62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2" Nov 03 20:59:16.485572 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:59:16.485548 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2\": container with ID starting with 62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2 not found: ID does not exist" containerID="62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2" Nov 03 20:59:16.485645 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.485582 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2"} err="failed to get container status \"62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2\": rpc error: code = NotFound desc = could not find container \"62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2\": container with ID starting with 62b5b69890b37164dc2099643256eee3d35e78574ce4de0bbb100187b4ad70a2 not found: ID does not exist" Nov 03 20:59:16.595673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.595642 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:16.604902 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.604875 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-socket\") pod \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " Nov 03 20:59:16.605025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.604958 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-podinfo\") pod \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " Nov 03 20:59:16.605025 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605022 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-certs\") pod \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " Nov 03 20:59:16.605117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605048 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-data\") pod \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " Nov 03 20:59:16.605117 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605100 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-token\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-token\") pod \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " Nov 03 20:59:16.605227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605113 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-socket" pod "d26ab542-92c0-4424-8dd3-d74dc2839bbb" (UID: "d26ab542-92c0-4424-8dd3-d74dc2839bbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:16.605227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605135 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-credential-socket\") pod \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " Nov 03 20:59:16.605227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605162 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-kube-api-access-jj2kr\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-kube-api-access-jj2kr\") pod \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " Nov 03 20:59:16.605227 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605202 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-envoy\") pod \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " Nov 03 20:59:16.605456 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605237 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istiod-ca-cert\") pod \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\" (UID: \"d26ab542-92c0-4424-8dd3-d74dc2839bbb\") " Nov 03 20:59:16.605456 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605350 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-data" pod "d26ab542-92c0-4424-8dd3-d74dc2839bbb" (UID: "d26ab542-92c0-4424-8dd3-d74dc2839bbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:16.605456 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605369 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-certs" pod "d26ab542-92c0-4424-8dd3-d74dc2839bbb" (UID: "d26ab542-92c0-4424-8dd3-d74dc2839bbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:16.605456 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605417 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-credential-socket" pod "d26ab542-92c0-4424-8dd3-d74dc2839bbb" (UID: "d26ab542-92c0-4424-8dd3-d74dc2839bbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:16.605746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605724 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istiod-ca-cert" pod "d26ab542-92c0-4424-8dd3-d74dc2839bbb" (UID: "d26ab542-92c0-4424-8dd3-d74dc2839bbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:16.605988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605959 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:16.605988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605976 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:16.605988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.605988 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:16.606182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.606003 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:16.606182 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.606019 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:16.607378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.607356 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-token" pod "d26ab542-92c0-4424-8dd3-d74dc2839bbb" (UID: "d26ab542-92c0-4424-8dd3-d74dc2839bbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:16.607378 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.607362 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-envoy" pod "d26ab542-92c0-4424-8dd3-d74dc2839bbb" (UID: "d26ab542-92c0-4424-8dd3-d74dc2839bbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:16.607610 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.607592 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-podinfo" pod "d26ab542-92c0-4424-8dd3-d74dc2839bbb" (UID: "d26ab542-92c0-4424-8dd3-d74dc2839bbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:16.607651 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.607613 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-kube-api-access-jj2kr" pod "d26ab542-92c0-4424-8dd3-d74dc2839bbb" (UID: "d26ab542-92c0-4424-8dd3-d74dc2839bbb"). InnerVolumeSpecName "kube-api-access-jj2kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:16.708850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.708812 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:16.708850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.708848 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:16.708850 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.708864 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2kr\" (UniqueName: \"kubernetes.io/projected/d26ab542-92c0-4424-8dd3-d74dc2839bbb-kube-api-access-jj2kr\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:16.709162 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.708877 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26ab542-92c0-4424-8dd3-d74dc2839bbb-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:16.796192 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.796161 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg"] Nov 03 20:59:16.799557 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:16.799533 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3c25bf-openshift-default-7d78cb5b9c-wf2rg"] Nov 03 20:59:17.365505 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:17.363879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:17.534295 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:59:17.534251 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:59:18.075693 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:18.075649 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:18.277040 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:18.277005 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:18.683582 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:18.683548 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" path="/var/lib/kubelet/pods/d26ab542-92c0-4424-8dd3-d74dc2839bbb/volumes" Nov 03 20:59:18.796463 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:18.796423 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8"] Nov 03 20:59:18.796695 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:18.796668 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" podUID="72411ecc-c8d4-4366-9838-0670268e3d05" containerName="maas-api" containerID="cri-o://b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23" gracePeriod=30 Nov 03 20:59:18.814911 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:18.814882 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:18.960112 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:18.960023 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:19.159986 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.159958 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:59:19.172281 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.172137 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:19.262316 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.262282 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72411ecc-c8d4-4366-9838-0670268e3d05-kube-api-access-75kls\" (UniqueName: \"kubernetes.io/projected/72411ecc-c8d4-4366-9838-0670268e3d05-kube-api-access-75kls\") pod \"72411ecc-c8d4-4366-9838-0670268e3d05\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " Nov 03 20:59:19.266482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.264671 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72411ecc-c8d4-4366-9838-0670268e3d05-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72411ecc-c8d4-4366-9838-0670268e3d05-openshift-service-ca\") pod \"72411ecc-c8d4-4366-9838-0670268e3d05\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " Nov 03 20:59:19.266482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.264737 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-maas-api-tls\") pod \"72411ecc-c8d4-4366-9838-0670268e3d05\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " Nov 03 20:59:19.266482 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.264876 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-metrics-certs\") pod \"72411ecc-c8d4-4366-9838-0670268e3d05\" (UID: \"72411ecc-c8d4-4366-9838-0670268e3d05\") " Nov 03 20:59:19.273567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.270531 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72411ecc-c8d4-4366-9838-0670268e3d05-kube-api-access-75kls" pod "72411ecc-c8d4-4366-9838-0670268e3d05" (UID: "72411ecc-c8d4-4366-9838-0670268e3d05"). InnerVolumeSpecName "kube-api-access-75kls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:19.273567 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.270907 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72411ecc-c8d4-4366-9838-0670268e3d05-openshift-service-ca" pod "72411ecc-c8d4-4366-9838-0670268e3d05" (UID: "72411ecc-c8d4-4366-9838-0670268e3d05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:19.290909 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.290876 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-maas-api-tls" pod "72411ecc-c8d4-4366-9838-0670268e3d05" (UID: "72411ecc-c8d4-4366-9838-0670268e3d05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:59:19.300151 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.297133 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-metrics-certs" pod "72411ecc-c8d4-4366-9838-0670268e3d05" (UID: "72411ecc-c8d4-4366-9838-0670268e3d05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:59:19.367996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.367961 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72411ecc-c8d4-4366-9838-0670268e3d05-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:19.367996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.367989 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:19.367996 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.367999 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72411ecc-c8d4-4366-9838-0670268e3d05-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:19.368222 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.368009 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75kls\" (UniqueName: \"kubernetes.io/projected/72411ecc-c8d4-4366-9838-0670268e3d05-kube-api-access-75kls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:19.392617 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.392548 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:19.498344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.498312 4711 generic.go:356] "Generic (PLEG): container finished" podID="72411ecc-c8d4-4366-9838-0670268e3d05" containerID="b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23" exitCode=0 Nov 03 20:59:19.498570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.498387 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" Nov 03 20:59:19.498570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.498404 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" event={"ID":"72411ecc-c8d4-4366-9838-0670268e3d05","Type":"ContainerDied","Data":"b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23"} Nov 03 20:59:19.498570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.498532 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8" event={"ID":"72411ecc-c8d4-4366-9838-0670268e3d05","Type":"ContainerDied","Data":"014ab7b503354707bac4e89e3007c3cf03cfe72808b1f843684c7e998cec35cb"} Nov 03 20:59:19.498570 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.498563 4711 scope.go:122] "RemoveContainer" containerID="b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23" Nov 03 20:59:19.509024 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.509008 4711 scope.go:122] "RemoveContainer" containerID="b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23" Nov 03 20:59:19.509303 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:59:19.509281 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23\": container with ID starting with b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23 not found: ID does not exist" containerID="b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23" Nov 03 20:59:19.509366 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.509310 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23"} err="failed to get container status \"b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23\": rpc error: code = NotFound desc = could not find container \"b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23\": container with ID starting with b6fee5a78dc0f64f3335e5b7257229bc377a149e3215eecaf20704b08387ca23 not found: ID does not exist" Nov 03 20:59:19.526023 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.525958 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8"] Nov 03 20:59:19.530355 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:19.530335 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-30e9de-66888c4bdc-dvqq8"] Nov 03 20:59:20.199369 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:20.199274 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:20.681284 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:20.681252 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72411ecc-c8d4-4366-9838-0670268e3d05" path="/var/lib/kubelet/pods/72411ecc-c8d4-4366-9838-0670268e3d05/volumes" Nov 03 20:59:21.005701 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:21.005629 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:21.683494 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:21.683406 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:22.435538 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:22.435504 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:23.113703 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:23.113672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:23.804116 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:23.804074 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:24.519988 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:24.519938 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:25.195835 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:25.195797 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:26.017097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:26.017062 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:26.217175 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:26.217136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:26.352327 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:26.352239 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:27.003483 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:27.000866 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:27.840171 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:27.840131 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:29.021669 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:29.021631 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:29.705367 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:29.705305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:30.415849 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:30.415793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:31.147894 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:31.147840 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:31.936525 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:31.936479 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:32.718244 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:32.718208 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:33.511016 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:33.510980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:34.384480 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:34.383030 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:35.135348 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:35.135303 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:35.809975 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:35.809934 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:36.691087 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:36.678063 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:36.986768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:36.986668 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:59:36.986768 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:36.986726 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:59:37.311725 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:37.311670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:38.017754 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:38.017722 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:38.715274 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:38.715193 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:38.742990 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:38.742957 4711 generic.go:356] "Generic (PLEG): container finished" podID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerID="8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02" exitCode=0 Nov 03 20:59:38.743347 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:38.743317 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" event={"ID":"159c4fe1-8080-4f38-b1d9-d67250329c9a","Type":"ContainerDied","Data":"8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02"} Nov 03 20:59:38.744032 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:38.744009 4711 scope.go:122] "RemoveContainer" containerID="8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02" Nov 03 20:59:38.746974 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:38.746929 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:59:38.747082 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:38.747034 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:59:38.747141 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:38.747080 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:59:39.362242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:39.362183 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:39.751851 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:39.751766 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" event={"ID":"159c4fe1-8080-4f38-b1d9-d67250329c9a","Type":"ContainerStarted","Data":"bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231"} Nov 03 20:59:40.021028 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:40.020990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:40.647001 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:40.646966 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:59:40.647406 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:40.647191 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:59:40.647406 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:40.647233 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 20:59:40.751673 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:40.751642 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:41.400890 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:41.400838 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:41.646551 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:41.646510 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:59:41.646737 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:41.646580 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 20:59:42.124546 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:42.124506 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:42.343331 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:42.343296 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:42.646628 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:42.646591 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:59:42.646810 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:42.646660 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 20:59:42.839344 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:42.839311 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:43.105937 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:43.105899 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:43.254524 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:43.254486 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:43.614085 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:43.614052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:43.646816 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:43.646783 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:59:43.646978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:43.646841 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 20:59:43.995118 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:43.995012 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:44.647157 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:44.647115 4711 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:59:44.647606 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:44.647183 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Nov 03 20:59:44.716935 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:44.716880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:45.404661 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:45.404627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:45.651097 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:45.651064 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:59:45.651622 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:45.651477 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:59:45.652128 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:45.652105 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 20:59:46.079605 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:46.079571 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:46.820169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:46.820131 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:47.533486 ip-10-0-142-5 kubenswrapper[4711]: E1103 20:59:47.533430 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 20:59:47.711404 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:47.711375 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:48.500514 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:48.500474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:48.610937 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:48.610898 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6"] Nov 03 20:59:48.629299 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:48.629268 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km"] Nov 03 20:59:48.740594 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:48.740557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:49.244750 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:49.244719 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:49.410548 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:49.410510 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:50.181646 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:50.181606 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:51.012622 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:51.012586 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:51.819242 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:51.815070 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:52.494615 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:52.494575 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:53.255202 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:53.255166 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:53.916340 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:53.916300 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:54.449542 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:54.449498 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:54.598978 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:54.598930 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:54.985973 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:54.985937 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:55.710083 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:55.710021 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:56.382746 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:56.382711 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:56.612665 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:56.612629 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:57.302169 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:57.302137 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:57.992511 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:57.992472 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:58.718813 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:58.718770 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:59.389532 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:59.389489 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 20:59:59.712595 ip-10-0-142-5 kubenswrapper[4711]: I1103 20:59:59.712506 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:00.077369 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.077332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:00.208995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.208963 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr"] Nov 03 21:00:00.209955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.209927 4711 cpu_manager.go:414] "containerMap: removing container" podUID="72411ecc-c8d4-4366-9838-0670268e3d05" containerName="maas-api" Nov 03 21:00:00.209955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.209952 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72411ecc-c8d4-4366-9838-0670268e3d05" containerName="maas-api" Nov 03 21:00:00.210159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.209980 4711 cpu_manager.go:414] "containerMap: removing container" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" Nov 03 21:00:00.210159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.209988 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" Nov 03 21:00:00.210159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210001 4711 cpu_manager.go:414] "containerMap: removing container" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" Nov 03 21:00:00.210159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210008 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" Nov 03 21:00:00.210159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210021 4711 cpu_manager.go:414] "containerMap: removing container" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" Nov 03 21:00:00.210159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210028 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" Nov 03 21:00:00.210159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210046 4711 cpu_manager.go:414] "containerMap: removing container" podUID="437f7610-bc66-4f26-9a3f-b751fb523aaf" containerName="maas-api" Nov 03 21:00:00.210159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210053 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437f7610-bc66-4f26-9a3f-b751fb523aaf" containerName="maas-api" Nov 03 21:00:00.212123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210248 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="72411ecc-c8d4-4366-9838-0670268e3d05" containerName="maas-api" Nov 03 21:00:00.212123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210265 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" Nov 03 21:00:00.212123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210277 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="774c317c-a849-4a4f-9921-18004a104b10" containerName="istio-proxy" Nov 03 21:00:00.212123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210289 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26ab542-92c0-4424-8dd3-d74dc2839bbb" containerName="istio-proxy" Nov 03 21:00:00.212123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.210301 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="437f7610-bc66-4f26-9a3f-b751fb523aaf" containerName="maas-api" Nov 03 21:00:00.217119 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.217079 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:00.218638 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.218611 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh"] Nov 03 21:00:00.224080 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.224036 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:00.231011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.227153 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr"] Nov 03 21:00:00.260587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.256632 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh"] Nov 03 21:00:00.260587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.259509 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7"] Nov 03 21:00:00.268966 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.268945 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:00:00.270648 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.270311 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr"] Nov 03 21:00:00.271351 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.271318 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:00.277167 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.277046 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:00.281394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.281374 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7"] Nov 03 21:00:00.292717 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.292689 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr"] Nov 03 21:00:00.292841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.292774 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7"] Nov 03 21:00:00.304252 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.304229 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7"] Nov 03 21:00:00.304395 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.304325 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:00.382865 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.382783 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/979e0405-c600-496d-8ca1-13f749ba72eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-kldwr\" (UID: \"979e0405-c600-496d-8ca1-13f749ba72eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:00.382865 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.382845 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dt96\" (UniqueName: \"kubernetes.io/projected/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-kube-api-access-4dt96\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7\" (UID: \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:00.383106 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.383006 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwdg\" (UniqueName: \"kubernetes.io/projected/68629248-0876-4c2b-afde-739373f15134-kube-api-access-lgwdg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7\" (UID: \"68629248-0876-4c2b-afde-739373f15134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:00.383106 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.383061 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7\" (UID: \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:00.383196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.383142 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m287p\" (UniqueName: \"kubernetes.io/projected/7b75931b-bb2c-4415-9785-8bcf16418c61-kube-api-access-m287p\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr\" (UID: \"7b75931b-bb2c-4415-9785-8bcf16418c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:00.383239 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.383196 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgsh2\" (UniqueName: \"kubernetes.io/projected/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-kube-api-access-hgsh2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh\" (UID: \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:00.383289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.383246 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh\" (UID: \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:00.383289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.383268 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68629248-0876-4c2b-afde-739373f15134-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7\" (UID: \"68629248-0876-4c2b-afde-739373f15134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:00.383357 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.383332 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75931b-bb2c-4415-9785-8bcf16418c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr\" (UID: \"7b75931b-bb2c-4415-9785-8bcf16418c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:00.383393 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.383387 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5fc\" (UniqueName: \"kubernetes.io/projected/979e0405-c600-496d-8ca1-13f749ba72eb-kube-api-access-bk5fc\") pod \"maas-api-key-cleanup-29895660-kldwr\" (UID: \"979e0405-c600-496d-8ca1-13f749ba72eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:00.485243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485212 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwdg\" (UniqueName: \"kubernetes.io/projected/68629248-0876-4c2b-afde-739373f15134-kube-api-access-lgwdg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7\" (UID: \"68629248-0876-4c2b-afde-739373f15134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:00.485408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485256 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7\" (UID: \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:00.485408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485386 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m287p\" (UniqueName: \"kubernetes.io/projected/7b75931b-bb2c-4415-9785-8bcf16418c61-kube-api-access-m287p\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr\" (UID: \"7b75931b-bb2c-4415-9785-8bcf16418c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:00.485540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485491 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgsh2\" (UniqueName: \"kubernetes.io/projected/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-kube-api-access-hgsh2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh\" (UID: \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:00.485589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485570 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh\" (UID: \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:00.485646 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485606 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68629248-0876-4c2b-afde-739373f15134-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7\" (UID: \"68629248-0876-4c2b-afde-739373f15134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:00.485703 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485643 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75931b-bb2c-4415-9785-8bcf16418c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr\" (UID: \"7b75931b-bb2c-4415-9785-8bcf16418c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:00.485761 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485736 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5fc\" (UniqueName: \"kubernetes.io/projected/979e0405-c600-496d-8ca1-13f749ba72eb-kube-api-access-bk5fc\") pod \"maas-api-key-cleanup-29895660-kldwr\" (UID: \"979e0405-c600-496d-8ca1-13f749ba72eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:00.485873 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485830 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/979e0405-c600-496d-8ca1-13f749ba72eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-kldwr\" (UID: \"979e0405-c600-496d-8ca1-13f749ba72eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:00.485995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.485926 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dt96\" (UniqueName: \"kubernetes.io/projected/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-kube-api-access-4dt96\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7\" (UID: \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:00.486059 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.486033 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7\" (UID: \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:00.486398 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.486376 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh\" (UID: \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:00.486514 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.486433 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/979e0405-c600-496d-8ca1-13f749ba72eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-kldwr\" (UID: \"979e0405-c600-496d-8ca1-13f749ba72eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:00.486514 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.486435 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68629248-0876-4c2b-afde-739373f15134-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7\" (UID: \"68629248-0876-4c2b-afde-739373f15134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:00.486514 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.486436 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75931b-bb2c-4415-9785-8bcf16418c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr\" (UID: \"7b75931b-bb2c-4415-9785-8bcf16418c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:00.500978 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.500955 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5fc\" (UniqueName: \"kubernetes.io/projected/979e0405-c600-496d-8ca1-13f749ba72eb-kube-api-access-bk5fc\") pod \"maas-api-key-cleanup-29895660-kldwr\" (UID: \"979e0405-c600-496d-8ca1-13f749ba72eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:00.501118 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.501092 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dt96\" (UniqueName: \"kubernetes.io/projected/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-kube-api-access-4dt96\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7\" (UID: \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:00.501118 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.501111 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwdg\" (UniqueName: \"kubernetes.io/projected/68629248-0876-4c2b-afde-739373f15134-kube-api-access-lgwdg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7\" (UID: \"68629248-0876-4c2b-afde-739373f15134\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:00.501293 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.501271 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m287p\" (UniqueName: \"kubernetes.io/projected/7b75931b-bb2c-4415-9785-8bcf16418c61-kube-api-access-m287p\") pod \"maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr\" (UID: \"7b75931b-bb2c-4415-9785-8bcf16418c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:00.501348 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.501301 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgsh2\" (UniqueName: \"kubernetes.io/projected/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-kube-api-access-hgsh2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh\" (UID: \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:00.575002 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.574975 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:00.586572 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.586546 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:00.604480 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.603747 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:00.613172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.613139 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:00.621225 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.621207 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:00.887289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.887251 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh"] Nov 03 21:00:00.888891 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:00:00.888862 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbd2d74_56ec_4af8_8abb_9e8a10e04eaa.slice/crio-bbfd50794deb4cc6695abf114436910f42934c86027b1976e6b0b4b540d2c31e WatchSource:0}: Error finding container bbfd50794deb4cc6695abf114436910f42934c86027b1976e6b0b4b540d2c31e: Status 404 returned error can't find the container with id bbfd50794deb4cc6695abf114436910f42934c86027b1976e6b0b4b540d2c31e Nov 03 21:00:00.889658 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:00:00.889560 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979e0405_c600_496d_8ca1_13f749ba72eb.slice/crio-abdcb15b914d8d594453fc8d092c07bd9d3762a3bf61145d922d94922b2c5957 WatchSource:0}: Error finding container abdcb15b914d8d594453fc8d092c07bd9d3762a3bf61145d922d94922b2c5957: Status 404 returned error can't find the container with id abdcb15b914d8d594453fc8d092c07bd9d3762a3bf61145d922d94922b2c5957 Nov 03 21:00:00.892427 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.892400 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr"] Nov 03 21:00:00.938665 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:00:00.938623 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68629248_0876_4c2b_afde_739373f15134.slice/crio-9eb27ed37291f7f99b7966094040dc37b1441da9e083c6eccc62437d9bed680d WatchSource:0}: Error finding container 9eb27ed37291f7f99b7966094040dc37b1441da9e083c6eccc62437d9bed680d: Status 404 returned error can't find the container with id 9eb27ed37291f7f99b7966094040dc37b1441da9e083c6eccc62437d9bed680d Nov 03 21:00:00.938757 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.938737 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7"] Nov 03 21:00:00.942177 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:00:00.941852 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b75931b_bb2c_4415_9785_8bcf16418c61.slice/crio-e07f134aa56787a7b757300af2cfb940b418281292e42b62c75831ea50d6d3d6 WatchSource:0}: Error finding container e07f134aa56787a7b757300af2cfb940b418281292e42b62c75831ea50d6d3d6: Status 404 returned error can't find the container with id e07f134aa56787a7b757300af2cfb940b418281292e42b62c75831ea50d6d3d6 Nov 03 21:00:00.946005 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.942733 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr"] Nov 03 21:00:00.963062 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.963031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:00.974922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.974891 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" event={"ID":"7b75931b-bb2c-4415-9785-8bcf16418c61","Type":"ContainerStarted","Data":"e07f134aa56787a7b757300af2cfb940b418281292e42b62c75831ea50d6d3d6"} Nov 03 21:00:00.976143 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.976118 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" event={"ID":"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa","Type":"ContainerStarted","Data":"bbfd50794deb4cc6695abf114436910f42934c86027b1976e6b0b4b540d2c31e"} Nov 03 21:00:00.977669 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.977643 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" event={"ID":"68629248-0876-4c2b-afde-739373f15134","Type":"ContainerStarted","Data":"9eb27ed37291f7f99b7966094040dc37b1441da9e083c6eccc62437d9bed680d"} Nov 03 21:00:00.979758 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.979737 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" event={"ID":"979e0405-c600-496d-8ca1-13f749ba72eb","Type":"ContainerStarted","Data":"abdcb15b914d8d594453fc8d092c07bd9d3762a3bf61145d922d94922b2c5957"} Nov 03 21:00:00.982154 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:00.982136 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7"] Nov 03 21:00:01.090412 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.090386 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:01.982775 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.982745 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:01.988868 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.988845 4711 generic.go:356] "Generic (PLEG): container finished" podID="979e0405-c600-496d-8ca1-13f749ba72eb" containerID="8d007dc16a94eba40b27cd581780e132b94c37fbdd1d176f2162ba6326a80560" exitCode=60 Nov 03 21:00:01.989004 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.988928 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" event={"ID":"979e0405-c600-496d-8ca1-13f749ba72eb","Type":"ContainerDied","Data":"8d007dc16a94eba40b27cd581780e132b94c37fbdd1d176f2162ba6326a80560"} Nov 03 21:00:01.989238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.989219 4711 scope.go:122] "RemoveContainer" containerID="8d007dc16a94eba40b27cd581780e132b94c37fbdd1d176f2162ba6326a80560" Nov 03 21:00:01.990874 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.990817 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" event={"ID":"7b75931b-bb2c-4415-9785-8bcf16418c61","Type":"ContainerStarted","Data":"9ce5aa3e124315d3cd9627d2ecb3f3775a94680f42ab93d5ea005ad5a1924489"} Nov 03 21:00:01.993411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.993377 4711 generic.go:356] "Generic (PLEG): container finished" podID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerID="5622f98f75cd6a7e07816b69d63a2e16ed33146eae16f143b3466d588121a46a" exitCode=60 Nov 03 21:00:01.993586 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.993424 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" event={"ID":"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa","Type":"ContainerDied","Data":"5622f98f75cd6a7e07816b69d63a2e16ed33146eae16f143b3466d588121a46a"} Nov 03 21:00:01.993835 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.993745 4711 scope.go:122] "RemoveContainer" containerID="5622f98f75cd6a7e07816b69d63a2e16ed33146eae16f143b3466d588121a46a" Nov 03 21:00:01.995488 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.995462 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" event={"ID":"68629248-0876-4c2b-afde-739373f15134","Type":"ContainerStarted","Data":"f130da8bf5c8224786bd97ada8eddfd98a1f483e7607d24006f6a51d748dc2b5"} Nov 03 21:00:01.995756 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.995743 4711 scope.go:122] "RemoveContainer" containerID="f130da8bf5c8224786bd97ada8eddfd98a1f483e7607d24006f6a51d748dc2b5" Nov 03 21:00:01.997290 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.997267 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" event={"ID":"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc","Type":"ContainerStarted","Data":"4cbf5fc9ac40ac2d863ba7b69004b79dacd709bb2be86ae128a40fa45ee79cd5"} Nov 03 21:00:01.997411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:01.997312 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" event={"ID":"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc","Type":"ContainerStarted","Data":"65933fdca3b24b72458358522bdddb5051d85c4fdb06f4a02754697e0c04a999"} Nov 03 21:00:02.072849 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:02.072803 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" podStartSLOduration=2.072791362 podStartE2EDuration="2.072791362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.067956786 +0000 UTC m=+3505.518062833" watchObservedRunningTime="2026-11-03 21:00:02.072791362 +0000 UTC m=+3505.522897406" Nov 03 21:00:02.091076 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:02.091018 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" podStartSLOduration=2.090990077 podStartE2EDuration="2.090990077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.086098435 +0000 UTC m=+3505.536204481" watchObservedRunningTime="2026-11-03 21:00:02.090990077 +0000 UTC m=+3505.541096125" Nov 03 21:00:02.714311 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:02.714275 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:03.005152 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.005120 4711 generic.go:356] "Generic (PLEG): container finished" podID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerID="9ce5aa3e124315d3cd9627d2ecb3f3775a94680f42ab93d5ea005ad5a1924489" exitCode=60 Nov 03 21:00:03.005335 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.005170 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" event={"ID":"7b75931b-bb2c-4415-9785-8bcf16418c61","Type":"ContainerDied","Data":"9ce5aa3e124315d3cd9627d2ecb3f3775a94680f42ab93d5ea005ad5a1924489"} Nov 03 21:00:03.005565 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.005550 4711 scope.go:122] "RemoveContainer" containerID="9ce5aa3e124315d3cd9627d2ecb3f3775a94680f42ab93d5ea005ad5a1924489" Nov 03 21:00:03.007312 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.007291 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" event={"ID":"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa","Type":"ContainerStarted","Data":"95df2de63a1c556500662c5cc723897e0cd768899f9f35a50fd871f195992041"} Nov 03 21:00:03.009235 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.009213 4711 generic.go:356] "Generic (PLEG): container finished" podID="68629248-0876-4c2b-afde-739373f15134" containerID="f130da8bf5c8224786bd97ada8eddfd98a1f483e7607d24006f6a51d748dc2b5" exitCode=60 Nov 03 21:00:03.009410 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.009310 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" event={"ID":"68629248-0876-4c2b-afde-739373f15134","Type":"ContainerDied","Data":"f130da8bf5c8224786bd97ada8eddfd98a1f483e7607d24006f6a51d748dc2b5"} Nov 03 21:00:03.009410 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.009350 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" event={"ID":"68629248-0876-4c2b-afde-739373f15134","Type":"ContainerStarted","Data":"30599e3b6399ebb94ac52b902db529d457d3d6bcef116ab1da2424b232138152"} Nov 03 21:00:03.011157 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.011135 4711 generic.go:356] "Generic (PLEG): container finished" podID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerID="4cbf5fc9ac40ac2d863ba7b69004b79dacd709bb2be86ae128a40fa45ee79cd5" exitCode=60 Nov 03 21:00:03.011256 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.011163 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" event={"ID":"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc","Type":"ContainerDied","Data":"4cbf5fc9ac40ac2d863ba7b69004b79dacd709bb2be86ae128a40fa45ee79cd5"} Nov 03 21:00:03.011418 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.011400 4711 scope.go:122] "RemoveContainer" containerID="4cbf5fc9ac40ac2d863ba7b69004b79dacd709bb2be86ae128a40fa45ee79cd5" Nov 03 21:00:03.013283 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.013265 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" event={"ID":"979e0405-c600-496d-8ca1-13f749ba72eb","Type":"ContainerStarted","Data":"824c1e94238eeffafa26ba715b9386d99809ab864c166a41aedaa1c8c46601e2"} Nov 03 21:00:03.064287 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.064247 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" podStartSLOduration=3.064233846 podStartE2EDuration="3.064233846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.063843144 +0000 UTC m=+3506.513949191" watchObservedRunningTime="2026-11-03 21:00:03.064233846 +0000 UTC m=+3506.514339891" Nov 03 21:00:03.085189 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.085137 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" podStartSLOduration=3.08512439 podStartE2EDuration="3.08512439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.081421878 +0000 UTC m=+3506.531527939" watchObservedRunningTime="2026-11-03 21:00:03.08512439 +0000 UTC m=+3506.535230436" Nov 03 21:00:03.102132 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.102093 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" podStartSLOduration=3.102080691 podStartE2EDuration="3.102080691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:03.099376328 +0000 UTC m=+3506.549482375" watchObservedRunningTime="2026-11-03 21:00:03.102080691 +0000 UTC m=+3506.552186737" Nov 03 21:00:03.428252 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.428212 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:03.632555 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.632431 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" containerID="cri-o://59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3" gracePeriod=45 Nov 03 21:00:03.652709 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.652260 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" containerID="cri-o://3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca" gracePeriod=45 Nov 03 21:00:03.957082 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.957048 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 21:00:03.964586 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.964561 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57dadd50-7763-424a-91f3-4949d9a7bc79-kube-api-access-9s946\" (UniqueName: \"kubernetes.io/projected/57dadd50-7763-424a-91f3-4949d9a7bc79-kube-api-access-9s946\") pod \"57dadd50-7763-424a-91f3-4949d9a7bc79\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " Nov 03 21:00:03.964698 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.964649 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-tmp-dir\") pod \"57dadd50-7763-424a-91f3-4949d9a7bc79\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " Nov 03 21:00:03.964928 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.964901 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-tmp-dir" pod "57dadd50-7763-424a-91f3-4949d9a7bc79" (UID: "57dadd50-7763-424a-91f3-4949d9a7bc79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:03.965243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.965218 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:03.966326 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.966303 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 21:00:03.966893 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:03.966869 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dadd50-7763-424a-91f3-4949d9a7bc79-kube-api-access-9s946" pod "57dadd50-7763-424a-91f3-4949d9a7bc79" (UID: "57dadd50-7763-424a-91f3-4949d9a7bc79"). InnerVolumeSpecName "kube-api-access-9s946". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:04.028935 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.028901 4711 generic.go:356] "Generic (PLEG): container finished" podID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerID="3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca" exitCode=0 Nov 03 21:00:04.029129 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.029016 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" Nov 03 21:00:04.029129 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.029017 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" event={"ID":"4cf4fae2-dcad-47e2-ae65-d698bb535dab","Type":"ContainerDied","Data":"3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca"} Nov 03 21:00:04.029129 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.029072 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" event={"ID":"4cf4fae2-dcad-47e2-ae65-d698bb535dab","Type":"ContainerDied","Data":"9626eb4e6ffb952392289b8d142d17381635fc12dcfa070236db8b3f2066bfdc"} Nov 03 21:00:04.029129 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.029101 4711 scope.go:122] "RemoveContainer" containerID="3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca" Nov 03 21:00:04.031862 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.031836 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:04.034059 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.034032 4711 generic.go:356] "Generic (PLEG): container finished" podID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerID="9f411d370bc50d75601b4e9bc105d8e2f29bd3957232c7e73cc37f1b6756b6f2" exitCode=60 Nov 03 21:00:04.034154 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.034103 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" event={"ID":"7b75931b-bb2c-4415-9785-8bcf16418c61","Type":"ContainerDied","Data":"9f411d370bc50d75601b4e9bc105d8e2f29bd3957232c7e73cc37f1b6756b6f2"} Nov 03 21:00:04.034640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.034619 4711 scope.go:122] "RemoveContainer" containerID="9f411d370bc50d75601b4e9bc105d8e2f29bd3957232c7e73cc37f1b6756b6f2" Nov 03 21:00:04.035046 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:04.035022 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr_odh-ai-gateway-infra(7b75931b-bb2c-4415-9785-8bcf16418c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" Nov 03 21:00:04.037910 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.037880 4711 generic.go:356] "Generic (PLEG): container finished" podID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerID="95df2de63a1c556500662c5cc723897e0cd768899f9f35a50fd871f195992041" exitCode=60 Nov 03 21:00:04.038007 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.037944 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" event={"ID":"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa","Type":"ContainerDied","Data":"95df2de63a1c556500662c5cc723897e0cd768899f9f35a50fd871f195992041"} Nov 03 21:00:04.038239 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.038224 4711 scope.go:122] "RemoveContainer" containerID="95df2de63a1c556500662c5cc723897e0cd768899f9f35a50fd871f195992041" Nov 03 21:00:04.038566 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:04.038548 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh_odh-ai-gateway-infra(9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" Nov 03 21:00:04.041621 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.041596 4711 generic.go:356] "Generic (PLEG): container finished" podID="68629248-0876-4c2b-afde-739373f15134" containerID="30599e3b6399ebb94ac52b902db529d457d3d6bcef116ab1da2424b232138152" exitCode=60 Nov 03 21:00:04.041722 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.041694 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" event={"ID":"68629248-0876-4c2b-afde-739373f15134","Type":"ContainerDied","Data":"30599e3b6399ebb94ac52b902db529d457d3d6bcef116ab1da2424b232138152"} Nov 03 21:00:04.041970 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.041951 4711 scope.go:122] "RemoveContainer" containerID="30599e3b6399ebb94ac52b902db529d457d3d6bcef116ab1da2424b232138152" Nov 03 21:00:04.042319 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:04.042291 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7_odh-ai-gateway-infra(68629248-0876-4c2b-afde-739373f15134)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" podUID="68629248-0876-4c2b-afde-739373f15134" Nov 03 21:00:04.045542 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.045507 4711 scope.go:122] "RemoveContainer" containerID="3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca" Nov 03 21:00:04.046100 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:04.046077 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca\": container with ID starting with 3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca not found: ID does not exist" containerID="3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca" Nov 03 21:00:04.046180 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.046110 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca"} err="failed to get container status \"3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca\": rpc error: code = NotFound desc = could not find container \"3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca\": container with ID starting with 3e6daeecc734e2f09a7d02b2fd2662484eca06a871c73681ecff52e69cc54fca not found: ID does not exist" Nov 03 21:00:04.046180 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.046132 4711 scope.go:122] "RemoveContainer" containerID="9ce5aa3e124315d3cd9627d2ecb3f3775a94680f42ab93d5ea005ad5a1924489" Nov 03 21:00:04.046180 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.046157 4711 generic.go:356] "Generic (PLEG): container finished" podID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerID="923e4d0a633438f100d6b589025a55fb9ad146a6ac708bef54c729fb33c0a2b9" exitCode=60 Nov 03 21:00:04.046287 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.046262 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" event={"ID":"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc","Type":"ContainerDied","Data":"923e4d0a633438f100d6b589025a55fb9ad146a6ac708bef54c729fb33c0a2b9"} Nov 03 21:00:04.046597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.046579 4711 scope.go:122] "RemoveContainer" containerID="923e4d0a633438f100d6b589025a55fb9ad146a6ac708bef54c729fb33c0a2b9" Nov 03 21:00:04.046951 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:04.046914 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7_odh-ai-gateway-infra(7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" Nov 03 21:00:04.047650 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.047631 4711 generic.go:356] "Generic (PLEG): container finished" podID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerID="59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3" exitCode=0 Nov 03 21:00:04.047730 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.047670 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" Nov 03 21:00:04.047730 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.047710 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" event={"ID":"57dadd50-7763-424a-91f3-4949d9a7bc79","Type":"ContainerDied","Data":"59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3"} Nov 03 21:00:04.047842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.047741 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6" event={"ID":"57dadd50-7763-424a-91f3-4949d9a7bc79","Type":"ContainerDied","Data":"4f961dbcd2183ea96fdd61104073edbffa3a1ef1f34137117408d783683147d6"} Nov 03 21:00:04.049544 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.049525 4711 generic.go:356] "Generic (PLEG): container finished" podID="979e0405-c600-496d-8ca1-13f749ba72eb" containerID="824c1e94238eeffafa26ba715b9386d99809ab864c166a41aedaa1c8c46601e2" exitCode=60 Nov 03 21:00:04.049632 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.049550 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" event={"ID":"979e0405-c600-496d-8ca1-13f749ba72eb","Type":"ContainerDied","Data":"824c1e94238eeffafa26ba715b9386d99809ab864c166a41aedaa1c8c46601e2"} Nov 03 21:00:04.049790 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.049777 4711 scope.go:122] "RemoveContainer" containerID="824c1e94238eeffafa26ba715b9386d99809ab864c166a41aedaa1c8c46601e2" Nov 03 21:00:04.050010 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:04.049994 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-kldwr_odh-ai-gateway-infra(979e0405-c600-496d-8ca1-13f749ba72eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" Nov 03 21:00:04.072821 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.072781 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-home\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-home\") pod \"57dadd50-7763-424a-91f3-4949d9a7bc79\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " Nov 03 21:00:04.072987 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.072874 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57dadd50-7763-424a-91f3-4949d9a7bc79-tls-certs\" (UniqueName: \"kubernetes.io/secret/57dadd50-7763-424a-91f3-4949d9a7bc79-tls-certs\") pod \"57dadd50-7763-424a-91f3-4949d9a7bc79\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " Nov 03 21:00:04.072987 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.072908 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-dshm\") pod \"57dadd50-7763-424a-91f3-4949d9a7bc79\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " Nov 03 21:00:04.073093 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.072983 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-model-cache\") pod \"57dadd50-7763-424a-91f3-4949d9a7bc79\" (UID: \"57dadd50-7763-424a-91f3-4949d9a7bc79\") " Nov 03 21:00:04.073093 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.073021 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-home" pod "57dadd50-7763-424a-91f3-4949d9a7bc79" (UID: "57dadd50-7763-424a-91f3-4949d9a7bc79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.073225 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.073204 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-model-cache" pod "57dadd50-7763-424a-91f3-4949d9a7bc79" (UID: "57dadd50-7763-424a-91f3-4949d9a7bc79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.075421 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.075342 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-dshm" pod "57dadd50-7763-424a-91f3-4949d9a7bc79" (UID: "57dadd50-7763-424a-91f3-4949d9a7bc79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.075891 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.075864 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57dadd50-7763-424a-91f3-4949d9a7bc79-tls-certs" pod "57dadd50-7763-424a-91f3-4949d9a7bc79" (UID: "57dadd50-7763-424a-91f3-4949d9a7bc79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:04.076755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.076731 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.076895 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.076770 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57dadd50-7763-424a-91f3-4949d9a7bc79-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.076895 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.076783 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.076895 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.076805 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57dadd50-7763-424a-91f3-4949d9a7bc79-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.076895 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.076819 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s946\" (UniqueName: \"kubernetes.io/projected/57dadd50-7763-424a-91f3-4949d9a7bc79-kube-api-access-9s946\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.084646 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.084630 4711 scope.go:122] "RemoveContainer" containerID="5622f98f75cd6a7e07816b69d63a2e16ed33146eae16f143b3466d588121a46a" Nov 03 21:00:04.098564 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.098539 4711 scope.go:122] "RemoveContainer" containerID="f130da8bf5c8224786bd97ada8eddfd98a1f483e7607d24006f6a51d748dc2b5" Nov 03 21:00:04.118218 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.118188 4711 scope.go:122] "RemoveContainer" containerID="4cbf5fc9ac40ac2d863ba7b69004b79dacd709bb2be86ae128a40fa45ee79cd5" Nov 03 21:00:04.133023 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.132999 4711 scope.go:122] "RemoveContainer" containerID="59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3" Nov 03 21:00:04.142016 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.141993 4711 scope.go:122] "RemoveContainer" containerID="59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3" Nov 03 21:00:04.142300 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:04.142277 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3\": container with ID starting with 59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3 not found: ID does not exist" containerID="59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3" Nov 03 21:00:04.142389 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.142312 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3"} err="failed to get container status \"59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3\": rpc error: code = NotFound desc = could not find container \"59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3\": container with ID starting with 59a38ac10a252b7d9b64755b454b334edb35dbc71c83a5df1be40945495433c3 not found: ID does not exist" Nov 03 21:00:04.142389 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.142336 4711 scope.go:122] "RemoveContainer" containerID="8d007dc16a94eba40b27cd581780e132b94c37fbdd1d176f2162ba6326a80560" Nov 03 21:00:04.177500 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.177415 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tmp-dir\") pod \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " Nov 03 21:00:04.177619 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.177600 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf4fae2-dcad-47e2-ae65-d698bb535dab-kube-api-access-sx9f9\" (UniqueName: \"kubernetes.io/projected/4cf4fae2-dcad-47e2-ae65-d698bb535dab-kube-api-access-sx9f9\") pod \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " Nov 03 21:00:04.177659 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.177626 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tmp-dir" pod "4cf4fae2-dcad-47e2-ae65-d698bb535dab" (UID: "4cf4fae2-dcad-47e2-ae65-d698bb535dab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.177659 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.177632 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-model-cache\") pod \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " Nov 03 21:00:04.177737 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.177723 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-dshm\") pod \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " Nov 03 21:00:04.177773 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.177761 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-home\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-home\") pod \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " Nov 03 21:00:04.177848 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.177828 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-model-cache" pod "4cf4fae2-dcad-47e2-ae65-d698bb535dab" (UID: "4cf4fae2-dcad-47e2-ae65-d698bb535dab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.177919 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.177901 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tls-certs\") pod \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\" (UID: \"4cf4fae2-dcad-47e2-ae65-d698bb535dab\") " Nov 03 21:00:04.178060 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.178041 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-home" pod "4cf4fae2-dcad-47e2-ae65-d698bb535dab" (UID: "4cf4fae2-dcad-47e2-ae65-d698bb535dab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.178401 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.178379 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.178495 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.178406 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.178495 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.178420 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.179960 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.179936 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tls-certs" pod "4cf4fae2-dcad-47e2-ae65-d698bb535dab" (UID: "4cf4fae2-dcad-47e2-ae65-d698bb535dab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:04.180138 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.180106 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-dshm" pod "4cf4fae2-dcad-47e2-ae65-d698bb535dab" (UID: "4cf4fae2-dcad-47e2-ae65-d698bb535dab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:04.180227 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.180114 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf4fae2-dcad-47e2-ae65-d698bb535dab-kube-api-access-sx9f9" pod "4cf4fae2-dcad-47e2-ae65-d698bb535dab" (UID: "4cf4fae2-dcad-47e2-ae65-d698bb535dab"). InnerVolumeSpecName "kube-api-access-sx9f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:04.279378 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.279344 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cf4fae2-dcad-47e2-ae65-d698bb535dab-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.279551 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.279381 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9f9\" (UniqueName: \"kubernetes.io/projected/4cf4fae2-dcad-47e2-ae65-d698bb535dab-kube-api-access-sx9f9\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.279551 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.279398 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cf4fae2-dcad-47e2-ae65-d698bb535dab-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:04.301564 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.301532 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:04.377057 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.377029 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km"] Nov 03 21:00:04.380507 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.380470 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km"] Nov 03 21:00:04.391897 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.391869 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6"] Nov 03 21:00:04.395884 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.395848 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-466f83/distinct-w4-466f83-kserve-6fccd744f4-gwrc6"] Nov 03 21:00:04.682136 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.682065 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" path="/var/lib/kubelet/pods/4cf4fae2-dcad-47e2-ae65-d698bb535dab/volumes" Nov 03 21:00:04.682875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.682842 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" path="/var/lib/kubelet/pods/57dadd50-7763-424a-91f3-4949d9a7bc79/volumes" Nov 03 21:00:04.960567 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:04.960476 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-466f83/simulator-w4-466f83-kserve-6498bf78fd-ph4km" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:05.059673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:05.059636 4711 scope.go:122] "RemoveContainer" containerID="9f411d370bc50d75601b4e9bc105d8e2f29bd3957232c7e73cc37f1b6756b6f2" Nov 03 21:00:05.060064 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:05.060031 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr_odh-ai-gateway-infra(7b75931b-bb2c-4415-9785-8bcf16418c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" Nov 03 21:00:05.061298 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:05.061274 4711 scope.go:122] "RemoveContainer" containerID="95df2de63a1c556500662c5cc723897e0cd768899f9f35a50fd871f195992041" Nov 03 21:00:05.061742 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:05.061713 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh_odh-ai-gateway-infra(9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" Nov 03 21:00:05.063136 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:05.063121 4711 scope.go:122] "RemoveContainer" containerID="30599e3b6399ebb94ac52b902db529d457d3d6bcef116ab1da2424b232138152" Nov 03 21:00:05.063362 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:05.063343 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7_odh-ai-gateway-infra(68629248-0876-4c2b-afde-739373f15134)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" podUID="68629248-0876-4c2b-afde-739373f15134" Nov 03 21:00:05.065664 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:05.065638 4711 scope.go:122] "RemoveContainer" containerID="923e4d0a633438f100d6b589025a55fb9ad146a6ac708bef54c729fb33c0a2b9" Nov 03 21:00:05.065860 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:05.065837 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7_odh-ai-gateway-infra(7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" Nov 03 21:00:05.068359 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:05.068342 4711 scope.go:122] "RemoveContainer" containerID="824c1e94238eeffafa26ba715b9386d99809ab864c166a41aedaa1c8c46601e2" Nov 03 21:00:05.068587 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:05.068567 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-kldwr_odh-ai-gateway-infra(979e0405-c600-496d-8ca1-13f749ba72eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" Nov 03 21:00:05.147336 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:05.146688 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:05.851993 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:05.851953 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:06.795398 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:06.795366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:07.685262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:07.685234 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:08.688860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:08.688825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:09.478469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:09.476405 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:10.197289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:10.197251 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:10.911296 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:10.911256 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:11.064698 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:11.064663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:11.694620 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:11.694581 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:12.393989 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:12.393952 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:13.088893 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:13.088841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:13.811154 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:13.811112 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:14.615355 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:14.615321 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:14.673565 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:14.673526 4711 scope.go:122] "RemoveContainer" containerID="95df2de63a1c556500662c5cc723897e0cd768899f9f35a50fd871f195992041" Nov 03 21:00:14.673758 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:14.673636 4711 scope.go:122] "RemoveContainer" containerID="30599e3b6399ebb94ac52b902db529d457d3d6bcef116ab1da2424b232138152" Nov 03 21:00:14.673832 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:14.673755 4711 scope.go:122] "RemoveContainer" containerID="923e4d0a633438f100d6b589025a55fb9ad146a6ac708bef54c729fb33c0a2b9" Nov 03 21:00:14.673892 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:14.673880 4711 scope.go:122] "RemoveContainer" containerID="824c1e94238eeffafa26ba715b9386d99809ab864c166a41aedaa1c8c46601e2" Nov 03 21:00:15.151596 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:15.151505 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" event={"ID":"979e0405-c600-496d-8ca1-13f749ba72eb","Type":"ContainerStarted","Data":"6999d4a73895aeead7170df8820bb158402fa018e1ec4cab1867d087b05078a0"} Nov 03 21:00:15.153563 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:15.153533 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" event={"ID":"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa","Type":"ContainerStarted","Data":"bd896ee443eed51c79c9f35dd58fa000acb7fdefd50891ecd5ef74ebbd20213c"} Nov 03 21:00:15.155339 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:15.155320 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" event={"ID":"68629248-0876-4c2b-afde-739373f15134","Type":"ContainerStarted","Data":"d7c54b86a065dedd387de226359f3468288275c25050951e06eab8790b95c94c"} Nov 03 21:00:15.396076 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:15.396037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:15.673485 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:15.673457 4711 scope.go:122] "RemoveContainer" containerID="9f411d370bc50d75601b4e9bc105d8e2f29bd3957232c7e73cc37f1b6756b6f2" Nov 03 21:00:16.115149 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.115118 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:16.171435 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.171402 4711 generic.go:356] "Generic (PLEG): container finished" podID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerID="bd896ee443eed51c79c9f35dd58fa000acb7fdefd50891ecd5ef74ebbd20213c" exitCode=60 Nov 03 21:00:16.171642 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.171492 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" event={"ID":"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa","Type":"ContainerDied","Data":"bd896ee443eed51c79c9f35dd58fa000acb7fdefd50891ecd5ef74ebbd20213c"} Nov 03 21:00:16.171642 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.171562 4711 scope.go:122] "RemoveContainer" containerID="95df2de63a1c556500662c5cc723897e0cd768899f9f35a50fd871f195992041" Nov 03 21:00:16.172015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.171996 4711 scope.go:122] "RemoveContainer" containerID="bd896ee443eed51c79c9f35dd58fa000acb7fdefd50891ecd5ef74ebbd20213c" Nov 03 21:00:16.172368 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:16.172341 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh_odh-ai-gateway-infra(9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" Nov 03 21:00:16.174284 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.174254 4711 generic.go:356] "Generic (PLEG): container finished" podID="68629248-0876-4c2b-afde-739373f15134" containerID="d7c54b86a065dedd387de226359f3468288275c25050951e06eab8790b95c94c" exitCode=60 Nov 03 21:00:16.174394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.174334 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" event={"ID":"68629248-0876-4c2b-afde-739373f15134","Type":"ContainerDied","Data":"d7c54b86a065dedd387de226359f3468288275c25050951e06eab8790b95c94c"} Nov 03 21:00:16.174791 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.174774 4711 scope.go:122] "RemoveContainer" containerID="d7c54b86a065dedd387de226359f3468288275c25050951e06eab8790b95c94c" Nov 03 21:00:16.175092 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:16.175069 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7_odh-ai-gateway-infra(68629248-0876-4c2b-afde-739373f15134)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" podUID="68629248-0876-4c2b-afde-739373f15134" Nov 03 21:00:16.176505 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.176482 4711 generic.go:356] "Generic (PLEG): container finished" podID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerID="6572a68d984cc218e5369ab234f90fc97e6f99c6f03cf4d58b43cd4246db9744" exitCode=60 Nov 03 21:00:16.176598 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.176565 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" event={"ID":"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc","Type":"ContainerDied","Data":"6572a68d984cc218e5369ab234f90fc97e6f99c6f03cf4d58b43cd4246db9744"} Nov 03 21:00:16.177279 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.177034 4711 scope.go:122] "RemoveContainer" containerID="6572a68d984cc218e5369ab234f90fc97e6f99c6f03cf4d58b43cd4246db9744" Nov 03 21:00:16.177396 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:16.177355 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7_odh-ai-gateway-infra(7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" Nov 03 21:00:16.180653 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.180348 4711 generic.go:356] "Generic (PLEG): container finished" podID="979e0405-c600-496d-8ca1-13f749ba72eb" containerID="6999d4a73895aeead7170df8820bb158402fa018e1ec4cab1867d087b05078a0" exitCode=60 Nov 03 21:00:16.180653 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.180403 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" event={"ID":"979e0405-c600-496d-8ca1-13f749ba72eb","Type":"ContainerDied","Data":"6999d4a73895aeead7170df8820bb158402fa018e1ec4cab1867d087b05078a0"} Nov 03 21:00:16.180813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.180790 4711 scope.go:122] "RemoveContainer" containerID="6999d4a73895aeead7170df8820bb158402fa018e1ec4cab1867d087b05078a0" Nov 03 21:00:16.181102 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:16.181079 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-kldwr_odh-ai-gateway-infra(979e0405-c600-496d-8ca1-13f749ba72eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" Nov 03 21:00:16.184016 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.183994 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr"] Nov 03 21:00:16.187761 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.187742 4711 scope.go:122] "RemoveContainer" containerID="30599e3b6399ebb94ac52b902db529d457d3d6bcef116ab1da2424b232138152" Nov 03 21:00:16.202009 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.201990 4711 scope.go:122] "RemoveContainer" containerID="923e4d0a633438f100d6b589025a55fb9ad146a6ac708bef54c729fb33c0a2b9" Nov 03 21:00:16.206490 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.206427 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7"] Nov 03 21:00:16.217845 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.217826 4711 scope.go:122] "RemoveContainer" containerID="824c1e94238eeffafa26ba715b9386d99809ab864c166a41aedaa1c8c46601e2" Nov 03 21:00:16.226634 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.226604 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh"] Nov 03 21:00:16.904846 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:16.901567 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:17.190191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.190116 4711 generic.go:356] "Generic (PLEG): container finished" podID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerID="90255f515879bb458bc87d8c37ce68d16f8d4574d4f9494342dd940916ef44f4" exitCode=60 Nov 03 21:00:17.190370 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.190188 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" event={"ID":"7b75931b-bb2c-4415-9785-8bcf16418c61","Type":"ContainerDied","Data":"90255f515879bb458bc87d8c37ce68d16f8d4574d4f9494342dd940916ef44f4"} Nov 03 21:00:17.190370 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.190248 4711 scope.go:122] "RemoveContainer" containerID="9f411d370bc50d75601b4e9bc105d8e2f29bd3957232c7e73cc37f1b6756b6f2" Nov 03 21:00:17.190683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.190662 4711 scope.go:122] "RemoveContainer" containerID="90255f515879bb458bc87d8c37ce68d16f8d4574d4f9494342dd940916ef44f4" Nov 03 21:00:17.191027 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:17.191003 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr_odh-ai-gateway-infra(7b75931b-bb2c-4415-9785-8bcf16418c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" Nov 03 21:00:17.259935 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.259686 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7"] Nov 03 21:00:17.395707 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.395680 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:17.484345 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.484312 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/979e0405-c600-496d-8ca1-13f749ba72eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/979e0405-c600-496d-8ca1-13f749ba72eb-maas-api-service-ca\") pod \"979e0405-c600-496d-8ca1-13f749ba72eb\" (UID: \"979e0405-c600-496d-8ca1-13f749ba72eb\") " Nov 03 21:00:17.484661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.484637 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979e0405-c600-496d-8ca1-13f749ba72eb-kube-api-access-bk5fc\" (UniqueName: \"kubernetes.io/projected/979e0405-c600-496d-8ca1-13f749ba72eb-kube-api-access-bk5fc\") pod \"979e0405-c600-496d-8ca1-13f749ba72eb\" (UID: \"979e0405-c600-496d-8ca1-13f749ba72eb\") " Nov 03 21:00:17.485613 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.485591 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:17.486504 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.486124 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979e0405-c600-496d-8ca1-13f749ba72eb-maas-api-service-ca" pod "979e0405-c600-496d-8ca1-13f749ba72eb" (UID: "979e0405-c600-496d-8ca1-13f749ba72eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:17.489777 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.489694 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979e0405-c600-496d-8ca1-13f749ba72eb-kube-api-access-bk5fc" pod "979e0405-c600-496d-8ca1-13f749ba72eb" (UID: "979e0405-c600-496d-8ca1-13f749ba72eb"). InnerVolumeSpecName "kube-api-access-bk5fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:17.499373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.499346 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:17.503869 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.503842 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:17.555114 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:17.555079 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:00:17.586931 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.586905 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68629248-0876-4c2b-afde-739373f15134-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68629248-0876-4c2b-afde-739373f15134-maas-api-service-ca\") pod \"68629248-0876-4c2b-afde-739373f15134\" (UID: \"68629248-0876-4c2b-afde-739373f15134\") " Nov 03 21:00:17.587079 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587014 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68629248-0876-4c2b-afde-739373f15134-kube-api-access-lgwdg\" (UniqueName: \"kubernetes.io/projected/68629248-0876-4c2b-afde-739373f15134-kube-api-access-lgwdg\") pod \"68629248-0876-4c2b-afde-739373f15134\" (UID: \"68629248-0876-4c2b-afde-739373f15134\") " Nov 03 21:00:17.587079 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587042 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-kube-api-access-hgsh2\" (UniqueName: \"kubernetes.io/projected/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-kube-api-access-hgsh2\") pod \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\" (UID: \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\") " Nov 03 21:00:17.587173 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587101 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-kube-api-access-4dt96\" (UniqueName: \"kubernetes.io/projected/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-kube-api-access-4dt96\") pod \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\" (UID: \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\") " Nov 03 21:00:17.587173 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587168 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-maas-api-service-ca\") pod \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\" (UID: \"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa\") " Nov 03 21:00:17.587286 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587235 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-maas-api-service-ca\") pod \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\" (UID: \"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc\") " Nov 03 21:00:17.587953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587360 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68629248-0876-4c2b-afde-739373f15134-maas-api-service-ca" pod "68629248-0876-4c2b-afde-739373f15134" (UID: "68629248-0876-4c2b-afde-739373f15134"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:17.587953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587712 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-maas-api-service-ca" pod "9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" (UID: "9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:17.587953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587839 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-maas-api-service-ca" pod "7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" (UID: "7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:17.587953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587906 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68629248-0876-4c2b-afde-739373f15134-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.587953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587917 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/979e0405-c600-496d-8ca1-13f749ba72eb-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.587953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587926 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk5fc\" (UniqueName: \"kubernetes.io/projected/979e0405-c600-496d-8ca1-13f749ba72eb-kube-api-access-bk5fc\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.587953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.587934 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.589666 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.589641 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68629248-0876-4c2b-afde-739373f15134-kube-api-access-lgwdg" pod "68629248-0876-4c2b-afde-739373f15134" (UID: "68629248-0876-4c2b-afde-739373f15134"). InnerVolumeSpecName "kube-api-access-lgwdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:17.589773 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.589693 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-kube-api-access-hgsh2" pod "9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" (UID: "9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa"). InnerVolumeSpecName "kube-api-access-hgsh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:17.589773 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.589720 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-kube-api-access-4dt96" pod "7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" (UID: "7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc"). InnerVolumeSpecName "kube-api-access-4dt96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:17.689435 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.689402 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.689435 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.689433 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgwdg\" (UniqueName: \"kubernetes.io/projected/68629248-0876-4c2b-afde-739373f15134-kube-api-access-lgwdg\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.689667 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.689477 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgsh2\" (UniqueName: \"kubernetes.io/projected/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa-kube-api-access-hgsh2\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.689667 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.689488 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dt96\" (UniqueName: \"kubernetes.io/projected/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc-kube-api-access-4dt96\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:17.706050 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:17.705970 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:18.205955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.205923 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" Nov 03 21:00:18.206471 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.205952 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr" event={"ID":"979e0405-c600-496d-8ca1-13f749ba72eb","Type":"ContainerDied","Data":"abdcb15b914d8d594453fc8d092c07bd9d3762a3bf61145d922d94922b2c5957"} Nov 03 21:00:18.206471 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.206017 4711 scope.go:122] "RemoveContainer" containerID="6999d4a73895aeead7170df8820bb158402fa018e1ec4cab1867d087b05078a0" Nov 03 21:00:18.209077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.209057 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" Nov 03 21:00:18.209196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.209098 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh" event={"ID":"9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa","Type":"ContainerDied","Data":"bbfd50794deb4cc6695abf114436910f42934c86027b1976e6b0b4b540d2c31e"} Nov 03 21:00:18.210617 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.210597 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" Nov 03 21:00:18.210737 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.210614 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7" event={"ID":"68629248-0876-4c2b-afde-739373f15134","Type":"ContainerDied","Data":"9eb27ed37291f7f99b7966094040dc37b1441da9e083c6eccc62437d9bed680d"} Nov 03 21:00:18.215560 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.215538 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" event={"ID":"7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc","Type":"ContainerDied","Data":"65933fdca3b24b72458358522bdddb5051d85c4fdb06f4a02754697e0c04a999"} Nov 03 21:00:18.215560 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.215555 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7" Nov 03 21:00:18.219262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.219244 4711 scope.go:122] "RemoveContainer" containerID="bd896ee443eed51c79c9f35dd58fa000acb7fdefd50891ecd5ef74ebbd20213c" Nov 03 21:00:18.226673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.226650 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr"] Nov 03 21:00:18.242087 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.242059 4711 scope.go:122] "RemoveContainer" containerID="d7c54b86a065dedd387de226359f3468288275c25050951e06eab8790b95c94c" Nov 03 21:00:18.265894 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.265872 4711 scope.go:122] "RemoveContainer" containerID="6572a68d984cc218e5369ab234f90fc97e6f99c6f03cf4d58b43cd4246db9744" Nov 03 21:00:18.273002 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.272974 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh"] Nov 03 21:00:18.281494 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.281470 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6e5139-29895660-s8nfh"] Nov 03 21:00:18.304569 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.304545 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr"] Nov 03 21:00:18.317109 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.317086 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-kldwr"] Nov 03 21:00:18.354593 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.354564 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7"] Nov 03 21:00:18.359242 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.359206 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-08a7bc-29895660-gvxg7"] Nov 03 21:00:18.368269 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.368242 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7"] Nov 03 21:00:18.371518 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.371479 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-63fde4-29895660-qs2x7"] Nov 03 21:00:18.375701 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.375681 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9"] Nov 03 21:00:18.375899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.375873 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" containerID="cri-o://6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c" gracePeriod=30 Nov 03 21:00:18.415802 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.415781 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:18.500831 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.500744 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b75931b-bb2c-4415-9785-8bcf16418c61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75931b-bb2c-4415-9785-8bcf16418c61-maas-api-service-ca\") pod \"7b75931b-bb2c-4415-9785-8bcf16418c61\" (UID: \"7b75931b-bb2c-4415-9785-8bcf16418c61\") " Nov 03 21:00:18.501016 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.500901 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b75931b-bb2c-4415-9785-8bcf16418c61-kube-api-access-m287p\" (UniqueName: \"kubernetes.io/projected/7b75931b-bb2c-4415-9785-8bcf16418c61-kube-api-access-m287p\") pod \"7b75931b-bb2c-4415-9785-8bcf16418c61\" (UID: \"7b75931b-bb2c-4415-9785-8bcf16418c61\") " Nov 03 21:00:18.501181 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.501152 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b75931b-bb2c-4415-9785-8bcf16418c61-maas-api-service-ca" pod "7b75931b-bb2c-4415-9785-8bcf16418c61" (UID: "7b75931b-bb2c-4415-9785-8bcf16418c61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:18.501522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.501502 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b75931b-bb2c-4415-9785-8bcf16418c61-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.502959 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.502937 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b75931b-bb2c-4415-9785-8bcf16418c61-kube-api-access-m287p" pod "7b75931b-bb2c-4415-9785-8bcf16418c61" (UID: "7b75931b-bb2c-4415-9785-8bcf16418c61"). InnerVolumeSpecName "kube-api-access-m287p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:18.602806 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.602777 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m287p\" (UniqueName: \"kubernetes.io/projected/7b75931b-bb2c-4415-9785-8bcf16418c61-kube-api-access-m287p\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.686227 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.686170 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68629248-0876-4c2b-afde-739373f15134" path="/var/lib/kubelet/pods/68629248-0876-4c2b-afde-739373f15134/volumes" Nov 03 21:00:18.686752 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.686735 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" path="/var/lib/kubelet/pods/7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc/volumes" Nov 03 21:00:18.687216 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.687199 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" path="/var/lib/kubelet/pods/979e0405-c600-496d-8ca1-13f749ba72eb/volumes" Nov 03 21:00:18.688054 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.688035 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" path="/var/lib/kubelet/pods/9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa/volumes" Nov 03 21:00:18.688371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:18.688350 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:19.229049 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:19.229009 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" event={"ID":"7b75931b-bb2c-4415-9785-8bcf16418c61","Type":"ContainerDied","Data":"e07f134aa56787a7b757300af2cfb940b418281292e42b62c75831ea50d6d3d6"} Nov 03 21:00:19.229049 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:19.229037 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr" Nov 03 21:00:19.229597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:19.229119 4711 scope.go:122] "RemoveContainer" containerID="90255f515879bb458bc87d8c37ce68d16f8d4574d4f9494342dd940916ef44f4" Nov 03 21:00:19.267784 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:19.260460 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr"] Nov 03 21:00:19.271706 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:19.271682 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b13a14-29895660-dbzwr"] Nov 03 21:00:19.509963 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:19.509935 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:20.473505 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:20.472992 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:20.685431 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:20.685394 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" path="/var/lib/kubelet/pods/7b75931b-bb2c-4415-9785-8bcf16418c61/volumes" Nov 03 21:00:21.353534 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:21.353483 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:00:21.353730 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:21.353544 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:00:21.538496 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:21.535956 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:22.445340 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:22.437759 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:23.328502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.316872 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:23.647174 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.647144 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 21:00:23.702893 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.702855 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-credential-socket\") pod \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " Nov 03 21:00:23.703062 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.702909 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istiod-ca-cert\") pod \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " Nov 03 21:00:23.703062 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.702990 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-envoy\") pod \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " Nov 03 21:00:23.703062 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703054 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-data\") pod \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " Nov 03 21:00:23.703262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703090 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-certs\") pod \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " Nov 03 21:00:23.703262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703125 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-podinfo\") pod \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " Nov 03 21:00:23.703262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703163 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-credential-socket" pod "b762bf30-1ddf-4c2c-8d45-c38173b09e5b" (UID: "b762bf30-1ddf-4c2c-8d45-c38173b09e5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:23.703262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703185 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-kube-api-access-n267p\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-kube-api-access-n267p\") pod \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " Nov 03 21:00:23.703262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703232 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-token\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-token\") pod \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " Nov 03 21:00:23.703637 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703300 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-socket\") pod \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\" (UID: \"b762bf30-1ddf-4c2c-8d45-c38173b09e5b\") " Nov 03 21:00:23.704197 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703703 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-certs" pod "b762bf30-1ddf-4c2c-8d45-c38173b09e5b" (UID: "b762bf30-1ddf-4c2c-8d45-c38173b09e5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:23.704197 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703724 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-data" pod "b762bf30-1ddf-4c2c-8d45-c38173b09e5b" (UID: "b762bf30-1ddf-4c2c-8d45-c38173b09e5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:23.704197 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.703919 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-socket" pod "b762bf30-1ddf-4c2c-8d45-c38173b09e5b" (UID: "b762bf30-1ddf-4c2c-8d45-c38173b09e5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:23.704426 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.704402 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istiod-ca-cert" pod "b762bf30-1ddf-4c2c-8d45-c38173b09e5b" (UID: "b762bf30-1ddf-4c2c-8d45-c38173b09e5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:23.704426 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.704414 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:23.704569 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.704471 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:23.704569 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.704490 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:23.704569 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.704508 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:23.705941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.705907 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-kube-api-access-n267p" pod "b762bf30-1ddf-4c2c-8d45-c38173b09e5b" (UID: "b762bf30-1ddf-4c2c-8d45-c38173b09e5b"). InnerVolumeSpecName "kube-api-access-n267p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:23.706751 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.706719 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-token" pod "b762bf30-1ddf-4c2c-8d45-c38173b09e5b" (UID: "b762bf30-1ddf-4c2c-8d45-c38173b09e5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:23.706884 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.706748 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-envoy" pod "b762bf30-1ddf-4c2c-8d45-c38173b09e5b" (UID: "b762bf30-1ddf-4c2c-8d45-c38173b09e5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:23.707162 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.707140 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-podinfo" pod "b762bf30-1ddf-4c2c-8d45-c38173b09e5b" (UID: "b762bf30-1ddf-4c2c-8d45-c38173b09e5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:00:23.806036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.806004 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:23.806036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.806032 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:23.806036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.806042 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:23.806270 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.806052 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:23.806270 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:23.806061 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n267p\" (UniqueName: \"kubernetes.io/projected/b762bf30-1ddf-4c2c-8d45-c38173b09e5b-kube-api-access-n267p\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:24.059432 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.059395 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:24.275941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.275909 4711 generic.go:356] "Generic (PLEG): container finished" podID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerID="6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c" exitCode=0 Nov 03 21:00:24.276128 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.275963 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" Nov 03 21:00:24.276128 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.275991 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" event={"ID":"b762bf30-1ddf-4c2c-8d45-c38173b09e5b","Type":"ContainerDied","Data":"6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c"} Nov 03 21:00:24.276128 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.276048 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9" event={"ID":"b762bf30-1ddf-4c2c-8d45-c38173b09e5b","Type":"ContainerDied","Data":"d89e07dc5943b9799d53fe9ed6fb4f63995e866a186906db0e9ef0bb57dd6ee7"} Nov 03 21:00:24.276128 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.276072 4711 scope.go:122] "RemoveContainer" containerID="6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c" Nov 03 21:00:24.288498 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.287212 4711 scope.go:122] "RemoveContainer" containerID="6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c" Nov 03 21:00:24.288957 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:24.288890 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c\": container with ID starting with 6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c not found: ID does not exist" containerID="6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c" Nov 03 21:00:24.288957 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.288932 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c"} err="failed to get container status \"6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c\": rpc error: code = NotFound desc = could not find container \"6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c\": container with ID starting with 6a8a6da8f84ef1c10bbd93dc24fbc1d5ca4a71097ac9bf9282cbbdc87e35065c not found: ID does not exist" Nov 03 21:00:24.303755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.303733 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9"] Nov 03 21:00:24.308665 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.308636 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-b6km9"] Nov 03 21:00:24.682066 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.682033 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" path="/var/lib/kubelet/pods/b762bf30-1ddf-4c2c-8d45-c38173b09e5b/volumes" Nov 03 21:00:24.927706 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:24.927670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:25.128140 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:25.128107 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:25.263718 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:25.263679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:25.523202 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:25.523170 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:25.624488 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:25.624417 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz"] Nov 03 21:00:25.624713 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:25.624685 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" podUID="8c4313d2-0e97-461e-9bd4-b698ce1daa2b" containerName="maas-api" containerID="cri-o://738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de" gracePeriod=30 Nov 03 21:00:25.724970 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:25.724931 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:25.994198 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:25.993830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:26.021248 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.020837 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 21:00:26.049109 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.045782 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-kube-api-access-qxbrw\" (UniqueName: \"kubernetes.io/projected/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-kube-api-access-qxbrw\") pod \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " Nov 03 21:00:26.049109 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.045883 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-openshift-service-ca\") pod \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " Nov 03 21:00:26.049109 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.045916 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-metrics-certs\") pod \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " Nov 03 21:00:26.049109 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.045950 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-maas-api-tls\") pod \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\" (UID: \"8c4313d2-0e97-461e-9bd4-b698ce1daa2b\") " Nov 03 21:00:26.049109 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.048677 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-openshift-service-ca" pod "8c4313d2-0e97-461e-9bd4-b698ce1daa2b" (UID: "8c4313d2-0e97-461e-9bd4-b698ce1daa2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:26.067929 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.051876 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-maas-api-tls" pod "8c4313d2-0e97-461e-9bd4-b698ce1daa2b" (UID: "8c4313d2-0e97-461e-9bd4-b698ce1daa2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:26.067929 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.051938 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-metrics-certs" pod "8c4313d2-0e97-461e-9bd4-b698ce1daa2b" (UID: "8c4313d2-0e97-461e-9bd4-b698ce1daa2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:26.067929 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.059702 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-kube-api-access-qxbrw" pod "8c4313d2-0e97-461e-9bd4-b698ce1daa2b" (UID: "8c4313d2-0e97-461e-9bd4-b698ce1daa2b"). InnerVolumeSpecName "kube-api-access-qxbrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:26.147675 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.147569 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxbrw\" (UniqueName: \"kubernetes.io/projected/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-kube-api-access-qxbrw\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:26.147675 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.147602 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:26.147675 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.147626 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:26.147675 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.147639 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c4313d2-0e97-461e-9bd4-b698ce1daa2b-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:26.248466 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.244180 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:26.293532 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.293494 4711 generic.go:356] "Generic (PLEG): container finished" podID="8c4313d2-0e97-461e-9bd4-b698ce1daa2b" containerID="738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de" exitCode=0 Nov 03 21:00:26.293532 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.293533 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" Nov 03 21:00:26.293794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.293549 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" event={"ID":"8c4313d2-0e97-461e-9bd4-b698ce1daa2b","Type":"ContainerDied","Data":"738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de"} Nov 03 21:00:26.293794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.293592 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" event={"ID":"8c4313d2-0e97-461e-9bd4-b698ce1daa2b","Type":"ContainerDied","Data":"2941c20b0222ade22ea817a98b42c2bf7688c2ae08b03ffaad0e6969096e0ca2"} Nov 03 21:00:26.293794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.293620 4711 scope.go:122] "RemoveContainer" containerID="738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de" Nov 03 21:00:26.314045 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.314002 4711 scope.go:122] "RemoveContainer" containerID="738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de" Nov 03 21:00:26.314536 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:26.314508 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de\": container with ID starting with 738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de not found: ID does not exist" containerID="738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de" Nov 03 21:00:26.314665 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.314540 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de"} err="failed to get container status \"738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de\": rpc error: code = NotFound desc = could not find container \"738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de\": container with ID starting with 738c47e974d794ffb9274a8ed8dc5277eaac708517a4bca110e4371c83d278de not found: ID does not exist" Nov 03 21:00:26.322596 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.322528 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz"] Nov 03 21:00:26.326370 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.326337 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz"] Nov 03 21:00:26.682808 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:26.682780 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4313d2-0e97-461e-9bd4-b698ce1daa2b" path="/var/lib/kubelet/pods/8c4313d2-0e97-461e-9bd4-b698ce1daa2b/volumes" Nov 03 21:00:27.108342 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:27.089377 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:27.246708 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:27.246655 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:27.885069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:27.885010 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:28.737225 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:28.737189 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:28.965293 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:28.965254 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-466f83-856d47f898-wxjkz" podUID="8c4313d2-0e97-461e-9bd4-b698ce1daa2b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.55:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:29.511519 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:29.511475 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:30.405172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:30.405130 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:31.300565 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:31.300530 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:32.062402 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:32.062366 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:32.752587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:32.752548 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:33.549030 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:33.548992 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:34.073671 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:34.073635 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:34.341916 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:34.341832 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:34.475203 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:34.475164 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:35.091967 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:35.091931 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:35.906162 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:35.906127 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:36.760652 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:36.744051 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:37.120780 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:37.120683 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:37.259295 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:37.259259 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:37.513364 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:37.512970 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:37.644540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:37.642880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:38.011725 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:38.010895 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2"] Nov 03 21:00:38.011725 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:38.011143 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" containerID="cri-o://372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a" gracePeriod=30 Nov 03 21:00:38.064024 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:38.063994 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:38.694993 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:38.694961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:39.752640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:39.752593 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:40.605097 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:40.602333 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:40.740788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:40.740747 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:41.444039 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:41.444002 4711 patch_prober.go:29] interesting pod/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:00:41.444534 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:41.444074 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:00:41.646971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:41.646934 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:41.774201 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:41.774171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:42.496081 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:42.496049 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:43.284282 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.284222 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 21:00:43.405042 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.405012 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:43.430524 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.428106 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-socket\") pod \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " Nov 03 21:00:43.430862 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.430831 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-podinfo\") pod \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " Nov 03 21:00:43.431007 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.430989 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-certs\") pod \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " Nov 03 21:00:43.431202 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.431185 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-credential-socket\") pod \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " Nov 03 21:00:43.431327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.431311 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-data\") pod \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " Nov 03 21:00:43.431416 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.431354 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-token\") pod \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " Nov 03 21:00:43.431416 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.431409 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istiod-ca-cert\") pod \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " Nov 03 21:00:43.431771 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.431502 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-kube-api-access-nzjjp\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-kube-api-access-nzjjp\") pod \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " Nov 03 21:00:43.431771 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.431534 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-envoy\") pod \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\" (UID: \"ff4a6ea8-1083-4baf-b1da-fd59c9c96224\") " Nov 03 21:00:43.432694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.432666 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-socket" pod "ff4a6ea8-1083-4baf-b1da-fd59c9c96224" (UID: "ff4a6ea8-1083-4baf-b1da-fd59c9c96224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.447500 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.442421 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-podinfo" pod "ff4a6ea8-1083-4baf-b1da-fd59c9c96224" (UID: "ff4a6ea8-1083-4baf-b1da-fd59c9c96224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:00:43.447500 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.446193 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-certs" pod "ff4a6ea8-1083-4baf-b1da-fd59c9c96224" (UID: "ff4a6ea8-1083-4baf-b1da-fd59c9c96224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.447500 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.446485 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-credential-socket" pod "ff4a6ea8-1083-4baf-b1da-fd59c9c96224" (UID: "ff4a6ea8-1083-4baf-b1da-fd59c9c96224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.447500 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.447363 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-data" pod "ff4a6ea8-1083-4baf-b1da-fd59c9c96224" (UID: "ff4a6ea8-1083-4baf-b1da-fd59c9c96224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.454228 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.450871 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istiod-ca-cert" pod "ff4a6ea8-1083-4baf-b1da-fd59c9c96224" (UID: "ff4a6ea8-1083-4baf-b1da-fd59c9c96224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:43.454228 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.451132 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-token" pod "ff4a6ea8-1083-4baf-b1da-fd59c9c96224" (UID: "ff4a6ea8-1083-4baf-b1da-fd59c9c96224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:43.454228 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.453948 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-envoy" pod "ff4a6ea8-1083-4baf-b1da-fd59c9c96224" (UID: "ff4a6ea8-1083-4baf-b1da-fd59c9c96224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:00:43.456181 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.455567 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-kube-api-access-nzjjp" pod "ff4a6ea8-1083-4baf-b1da-fd59c9c96224" (UID: "ff4a6ea8-1083-4baf-b1da-fd59c9c96224"). InnerVolumeSpecName "kube-api-access-nzjjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:43.473633 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.473605 4711 generic.go:356] "Generic (PLEG): container finished" podID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerID="372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a" exitCode=0 Nov 03 21:00:43.473786 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.473771 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" event={"ID":"ff4a6ea8-1083-4baf-b1da-fd59c9c96224","Type":"ContainerDied","Data":"372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a"} Nov 03 21:00:43.473865 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.473812 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" event={"ID":"ff4a6ea8-1083-4baf-b1da-fd59c9c96224","Type":"ContainerDied","Data":"6fd0a442de6e422f502033eb3eb1c1e2158b40e741d6639d3d5f4fca051e7952"} Nov 03 21:00:43.473865 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.473831 4711 scope.go:122] "RemoveContainer" containerID="372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a" Nov 03 21:00:43.473987 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.473951 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2" Nov 03 21:00:43.487244 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.487209 4711 scope.go:122] "RemoveContainer" containerID="1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2" Nov 03 21:00:43.497374 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.496897 4711 scope.go:122] "RemoveContainer" containerID="372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a" Nov 03 21:00:43.497374 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:43.497180 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a\": container with ID starting with 372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a not found: ID does not exist" containerID="372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a" Nov 03 21:00:43.497374 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.497213 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a"} err="failed to get container status \"372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a\": rpc error: code = NotFound desc = could not find container \"372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a\": container with ID starting with 372a350420d7a07c17f6ffec3613b5daefff36c112e416b9686e65a500ca615a not found: ID does not exist" Nov 03 21:00:43.497374 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.497233 4711 scope.go:122] "RemoveContainer" containerID="1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2" Nov 03 21:00:43.497995 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:43.497518 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2\": container with ID starting with 1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2 not found: ID does not exist" containerID="1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2" Nov 03 21:00:43.497995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.497545 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2"} err="failed to get container status \"1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2\": rpc error: code = NotFound desc = could not find container \"1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2\": container with ID starting with 1823f9fa0d5a2b7eab01c26464689dad858352a0467a805f0c886b080d8373f2 not found: ID does not exist" Nov 03 21:00:43.501115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.501093 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2"] Nov 03 21:00:43.508266 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.508244 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-466f83-openshift-default-5559865c55-x49b2"] Nov 03 21:00:43.533841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.533818 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.533841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.533841 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.533971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.533851 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.533971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.533859 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.533971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.533867 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzjjp\" (UniqueName: \"kubernetes.io/projected/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-kube-api-access-nzjjp\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.533971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.533876 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.533971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.533884 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.533971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.533892 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:43.533971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:43.533901 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff4a6ea8-1083-4baf-b1da-fd59c9c96224-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:44.117949 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:44.117908 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:44.680614 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:44.680570 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" path="/var/lib/kubelet/pods/ff4a6ea8-1083-4baf-b1da-fd59c9c96224/volumes" Nov 03 21:00:44.883820 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:44.883781 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:45.767015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:45.766966 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:46.540522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:46.539882 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:47.021700 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:47.021662 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:47.160589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:47.160539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:47.532276 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:47.532244 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:00:47.582600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:47.582555 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:48.400104 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:48.400068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:48.718689 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:48.718613 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:49.157133 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:49.157096 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:49.828972 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:49.828934 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:49.964585 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:49.964456 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:50.536065 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:50.536026 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:51.316981 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:51.316929 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:51.831988 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:51.831950 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d"] Nov 03 21:00:51.832232 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:51.832206 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" podUID="6f5fc815-027b-480b-bcda-a79f4d1bacd5" containerName="maas-api" containerID="cri-o://e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479" gracePeriod=30 Nov 03 21:00:51.913251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:51.913218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:52.189709 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.189682 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 21:00:52.269238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.269213 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5fc815-027b-480b-bcda-a79f4d1bacd5-kube-api-access-b6hpx\" (UniqueName: \"kubernetes.io/projected/6f5fc815-027b-480b-bcda-a79f4d1bacd5-kube-api-access-b6hpx\") pod \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " Nov 03 21:00:52.269413 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.269368 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5fc815-027b-480b-bcda-a79f4d1bacd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5fc815-027b-480b-bcda-a79f4d1bacd5-openshift-service-ca\") pod \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " Nov 03 21:00:52.269413 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.269400 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-metrics-certs\") pod \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " Nov 03 21:00:52.269516 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.269491 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-maas-api-tls\") pod \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\" (UID: \"6f5fc815-027b-480b-bcda-a79f4d1bacd5\") " Nov 03 21:00:52.269780 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.269736 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5fc815-027b-480b-bcda-a79f4d1bacd5-openshift-service-ca" pod "6f5fc815-027b-480b-bcda-a79f4d1bacd5" (UID: "6f5fc815-027b-480b-bcda-a79f4d1bacd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:52.269956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.269941 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f5fc815-027b-480b-bcda-a79f4d1bacd5-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.271267 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.271245 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5fc815-027b-480b-bcda-a79f4d1bacd5-kube-api-access-b6hpx" pod "6f5fc815-027b-480b-bcda-a79f4d1bacd5" (UID: "6f5fc815-027b-480b-bcda-a79f4d1bacd5"). InnerVolumeSpecName "kube-api-access-b6hpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:52.271372 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.271341 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-metrics-certs" pod "6f5fc815-027b-480b-bcda-a79f4d1bacd5" (UID: "6f5fc815-027b-480b-bcda-a79f4d1bacd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:52.271549 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.271532 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-maas-api-tls" pod "6f5fc815-027b-480b-bcda-a79f4d1bacd5" (UID: "6f5fc815-027b-480b-bcda-a79f4d1bacd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:00:52.372014 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.371973 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.372014 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.372016 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f5fc815-027b-480b-bcda-a79f4d1bacd5-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.372411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.372032 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6hpx\" (UniqueName: \"kubernetes.io/projected/6f5fc815-027b-480b-bcda-a79f4d1bacd5-kube-api-access-b6hpx\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:52.422975 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.422876 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:52.536013 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.535976 4711 generic.go:356] "Generic (PLEG): container finished" podID="6f5fc815-027b-480b-bcda-a79f4d1bacd5" containerID="e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479" exitCode=0 Nov 03 21:00:52.536188 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.536023 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" Nov 03 21:00:52.536188 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.536052 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" event={"ID":"6f5fc815-027b-480b-bcda-a79f4d1bacd5","Type":"ContainerDied","Data":"e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479"} Nov 03 21:00:52.536188 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.536102 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d" event={"ID":"6f5fc815-027b-480b-bcda-a79f4d1bacd5","Type":"ContainerDied","Data":"f69a63fb660fa5859000287790ebb6e954a4c0a3f4b396175be0931b5e697cc6"} Nov 03 21:00:52.536188 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.536123 4711 scope.go:122] "RemoveContainer" containerID="e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479" Nov 03 21:00:52.549170 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.548631 4711 scope.go:122] "RemoveContainer" containerID="e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479" Nov 03 21:00:52.549800 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:00:52.549420 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479\": container with ID starting with e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479 not found: ID does not exist" containerID="e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479" Nov 03 21:00:52.549800 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.549463 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479"} err="failed to get container status \"e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479\": rpc error: code = NotFound desc = could not find container \"e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479\": container with ID starting with e276eaae826e4663ed890ea7b4e9713f5d8751ecc461037da3e7aba4cbf61479 not found: ID does not exist" Nov 03 21:00:52.560419 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.560395 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d"] Nov 03 21:00:52.567787 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.567769 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-dce009-6cf6449f5f-6p75d"] Nov 03 21:00:52.590587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.590561 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:52.680587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:52.680531 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5fc815-027b-480b-bcda-a79f4d1bacd5" path="/var/lib/kubelet/pods/6f5fc815-027b-480b-bcda-a79f4d1bacd5/volumes" Nov 03 21:00:53.100151 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:53.100118 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:53.749830 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:53.749798 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:54.543639 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:54.543222 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:55.150625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:55.150581 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:55.285175 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:55.285143 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:55.409116 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:55.409033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:55.718941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:55.718850 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:55.970243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:55.970144 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:56.441924 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:56.441887 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:56.771607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:56.771565 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:56.906963 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:56.906927 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:57.198282 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:57.198196 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:57.937037 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:57.937002 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:58.600563 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:58.600530 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:59.312393 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:59.312355 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:00:59.752703 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:00:59.752606 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:00.009186 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:00.009153 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:00.721629 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:00.721578 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:01.031810 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:01.031769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:03.110843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.110812 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:03.269091 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.269060 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk"] Nov 03 21:01:03.270059 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270030 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerName="cleanup" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270059 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerName="cleanup" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270078 4711 cpu_manager.go:414] "containerMap: removing container" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" containerName="cleanup" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270086 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" containerName="cleanup" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270098 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270105 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270126 4711 cpu_manager.go:414] "containerMap: removing container" podUID="68629248-0876-4c2b-afde-739373f15134" containerName="cleanup" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270133 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68629248-0876-4c2b-afde-739373f15134" containerName="cleanup" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270149 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerName="cleanup" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270156 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerName="cleanup" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270166 4711 cpu_manager.go:414] "containerMap: removing container" podUID="68629248-0876-4c2b-afde-739373f15134" containerName="cleanup" Nov 03 21:01:03.270172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270173 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68629248-0876-4c2b-afde-739373f15134" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270187 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270194 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270206 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270212 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270227 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270234 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270243 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270250 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270262 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6f5fc815-027b-480b-bcda-a79f4d1bacd5" containerName="maas-api" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270269 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5fc815-027b-480b-bcda-a79f4d1bacd5" containerName="maas-api" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270285 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270292 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270314 4711 cpu_manager.go:414] "containerMap: removing container" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270320 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270329 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270337 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270353 4711 cpu_manager.go:414] "containerMap: removing container" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270361 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270396 4711 cpu_manager.go:414] "containerMap: removing container" podUID="8c4313d2-0e97-461e-9bd4-b698ce1daa2b" containerName="maas-api" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270405 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4313d2-0e97-461e-9bd4-b698ce1daa2b" containerName="maas-api" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270423 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270430 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270461 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerName="cleanup" Nov 03 21:01:03.270643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270470 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270676 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270692 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270702 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf4fae2-dcad-47e2-ae65-d698bb535dab" containerName="main" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270714 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="68629248-0876-4c2b-afde-739373f15134" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270727 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270739 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="68629248-0876-4c2b-afde-739373f15134" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270752 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270763 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270776 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270786 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4a6ea8-1083-4baf-b1da-fd59c9c96224" containerName="istio-proxy" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270796 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5fc815-027b-480b-bcda-a79f4d1bacd5" containerName="maas-api" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270804 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270815 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b762bf30-1ddf-4c2c-8d45-c38173b09e5b" containerName="istio-proxy" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270830 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270841 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6647a2-4cd2-4c12-9a23-bfa1b7d586fc" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270852 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="57dadd50-7763-424a-91f3-4949d9a7bc79" containerName="main" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270866 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270877 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4313d2-0e97-461e-9bd4-b698ce1daa2b" containerName="maas-api" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270887 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerName="cleanup" Nov 03 21:01:03.272371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.270895 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerName="cleanup" Nov 03 21:01:03.276900 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.276868 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.277683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.277655 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk"] Nov 03 21:01:03.283988 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.283965 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"kube-root-ca.crt\"" Nov 03 21:01:03.285196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.285176 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"openshift-service-ca.crt\"" Nov 03 21:01:03.287124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.285536 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"default-dockercfg-tzxvn\"" Nov 03 21:01:03.287124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.285620 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-99a8fe\"/\"sub-test-model-99a8fe-kserve-self-signed-certs\"" Nov 03 21:01:03.330885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.330851 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:03.469534 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.469430 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tls-certs\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.469874 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.469851 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tmp-dir\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.470087 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.470067 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-model-cache\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.470288 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.470266 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8nqq\" (UniqueName: \"kubernetes.io/projected/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-kube-api-access-j8nqq\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.470575 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.470552 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-home\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.470710 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.470696 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-dshm\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.572249 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.572215 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8nqq\" (UniqueName: \"kubernetes.io/projected/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-kube-api-access-j8nqq\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.572533 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.572318 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-home\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.572533 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.572343 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-dshm\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.572533 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.572379 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tls-certs\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.572533 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.572523 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tmp-dir\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.572807 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.572591 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-model-cache\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.572867 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.572818 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-home\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.572956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.572933 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-model-cache\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.573001 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.572971 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tmp-dir\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.574751 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.574728 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-dshm\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.574885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.574868 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tls-certs\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.584289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.584263 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8nqq\" (UniqueName: \"kubernetes.io/projected/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-kube-api-access-j8nqq\") pod \"sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.592195 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.592171 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:03.879406 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.879379 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk"] Nov 03 21:01:03.880640 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:01:03.880609 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4dfd3ff_7286_4fa6_83a1_4d249aa18d5a.slice/crio-514e3413cd0a1e98f640c8cf117aa9343eeab92eb469fc0f278566dce4842967 WatchSource:0}: Error finding container 514e3413cd0a1e98f640c8cf117aa9343eeab92eb469fc0f278566dce4842967: Status 404 returned error can't find the container with id 514e3413cd0a1e98f640c8cf117aa9343eeab92eb469fc0f278566dce4842967 Nov 03 21:01:03.920839 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:03.920817 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:04.300695 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:04.300661 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:04.433557 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:04.433513 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:04.562487 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:04.562388 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:04.632299 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:04.632264 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" event={"ID":"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a","Type":"ContainerStarted","Data":"4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59"} Nov 03 21:01:04.632299 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:04.632306 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" event={"ID":"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a","Type":"ContainerStarted","Data":"514e3413cd0a1e98f640c8cf117aa9343eeab92eb469fc0f278566dce4842967"} Nov 03 21:01:04.652894 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:04.652843 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" podStartSLOduration=1.652828626 podStartE2EDuration="1.652828626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:04.651592291 +0000 UTC m=+3568.101698333" watchObservedRunningTime="2026-11-03 21:01:04.652828626 +0000 UTC m=+3568.102934671" Nov 03 21:01:04.902010 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:04.898945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:05.592734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:05.592689 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:05.594074 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:05.594039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:06.272513 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:06.272474 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:07.014646 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:07.014614 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:07.812220 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:07.812179 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:08.594853 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:08.594820 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:09.428843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:09.428782 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:10.224513 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:10.224477 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:10.805575 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:10.805536 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:10.985910 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:10.985877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:11.900798 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:11.900762 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:12.713435 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:12.713396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:13.592515 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:13.592485 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:13.605166 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:13.605142 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:13.682150 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:13.682117 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:13.702956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:13.702928 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:01:14.420184 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:14.420153 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:14.757876 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:14.757841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:15.597459 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:15.597405 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:16.385310 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:16.385275 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:17.033169 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:17.033123 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:17.356186 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:17.356104 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:17.487191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:17.487146 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:17.536210 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:01:17.536166 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:01:17.706519 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:17.701969 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:17.881233 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:17.881177 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:18.233524 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:18.233484 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:18.699255 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:18.699213 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:19.033838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:19.033795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:19.438226 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:19.438140 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:20.302197 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:20.302158 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:20.983031 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:20.982978 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:21.660899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:21.660866 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:22.333864 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:22.333815 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:23.120092 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:23.120043 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:23.812379 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:23.812348 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:24.497491 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:24.497424 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:25.239394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:25.239361 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:25.371186 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:25.371137 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:25.997190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:25.997153 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:26.200622 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:26.200583 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:26.896832 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:26.896797 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:27.561912 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:27.561850 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:28.289361 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:28.289316 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:28.926223 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:28.926184 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:29.619991 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:29.619950 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:30.308665 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:30.308628 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:31.450646 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:31.450611 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:32.318775 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:32.318734 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:33.181283 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:33.181249 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:33.306997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:33.306960 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:33.914420 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:33.914388 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:34.593714 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:34.593679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:35.265044 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:35.264950 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:35.993332 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:35.993296 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:36.718168 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:36.718138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:37.082290 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:37.082240 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:37.847838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:37.847814 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:01:37.850964 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:37.850944 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:01:37.897763 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:37.897738 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:01:37.900635 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:37.900616 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:01:37.906982 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:37.906961 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:38.641391 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:38.641346 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:39.303918 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:39.303879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:39.443667 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:39.443631 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:39.816709 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:39.816585 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:40.138773 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:40.138676 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:40.311106 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:40.311056 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:40.943033 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:40.942987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:41.184904 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:41.184870 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:41.841902 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:41.841869 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:42.668841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:42.668806 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:43.361589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:43.361460 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:44.088768 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:44.088723 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:44.796095 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:44.796045 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:45.498641 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:45.498608 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:46.252417 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:46.252382 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:46.446750 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:46.446675 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:46.586121 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:46.586086 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:47.110902 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:47.110865 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:47.531823 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:01:47.531789 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:01:47.716893 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:47.716853 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:47.846414 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:47.846316 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:48.522043 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:48.522007 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:49.225788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:49.225745 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:49.933966 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:49.933928 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:50.614983 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:50.614952 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:51.485577 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:51.485539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:52.295561 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:52.295497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:52.988475 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:52.988417 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:53.712518 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:53.712477 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:54.369685 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:54.369646 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:54.514706 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:54.514672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:55.117838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:55.117798 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:55.892871 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:55.892830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:56.538829 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:56.538793 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:57.124095 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:57.124061 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:57.433985 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:57.433903 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:58.122084 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:58.122043 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:58.915924 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:58.915864 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:59.597399 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:59.597362 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:01:59.795661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:01:59.795630 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:00.290193 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:00.290156 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:00.770031 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:00.769996 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:00.900727 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:00.900693 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:01.088034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:01.087939 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:01.353774 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:01.353678 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:01.490786 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:01.490749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:01.757092 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:01.757061 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:02.231992 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:02.231910 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:02.380956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:02.380923 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:02.631433 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:02.631281 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:03.518397 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:03.518320 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:04.313053 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:04.313013 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:04.871558 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:04.871521 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:05.033222 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:05.033181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:05.323145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:05.323099 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:05.785745 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:05.785710 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:06.528485 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:06.528415 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:07.171645 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:07.171607 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr"] Nov 03 21:02:07.194219 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:07.194188 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq"] Nov 03 21:02:07.259052 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:07.259021 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:07.388193 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:07.388150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:07.981723 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:07.981690 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:08.110658 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:08.110625 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:08.786877 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:08.786841 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:09.528901 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:09.528867 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:10.401737 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:10.401705 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:11.308061 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:11.308022 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:12.022361 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:12.022328 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:12.324841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:12.324744 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:12.465416 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:12.465374 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:12.763297 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:12.763251 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:12.899770 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:12.899736 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:13.481296 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:13.481264 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m"] Nov 03 21:02:13.689402 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:13.689368 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:13.835481 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:13.832569 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:14.867526 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:14.867487 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:15.523206 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:15.523172 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:15.981435 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:15.981342 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:16.216641 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:16.216589 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:16.994583 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:16.994550 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:17.534487 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:02:17.534428 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:02:17.803009 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:17.802907 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:18.482541 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:18.482493 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" podUID="c64d74d3-b7bd-491e-84db-07c449ae1946" containerName="payload-processing" containerID="cri-o://bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa" gracePeriod=25 Nov 03 21:02:18.498411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:18.497529 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:18.759818 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:18.759786 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 21:02:18.845779 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:18.845736 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64d74d3-b7bd-491e-84db-07c449ae1946-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64d74d3-b7bd-491e-84db-07c449ae1946-plugins-config-volume\") pod \"c64d74d3-b7bd-491e-84db-07c449ae1946\" (UID: \"c64d74d3-b7bd-491e-84db-07c449ae1946\") " Nov 03 21:02:18.845967 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:18.845950 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64d74d3-b7bd-491e-84db-07c449ae1946-kube-api-access-j7rzj\" (UniqueName: \"kubernetes.io/projected/c64d74d3-b7bd-491e-84db-07c449ae1946-kube-api-access-j7rzj\") pod \"c64d74d3-b7bd-491e-84db-07c449ae1946\" (UID: \"c64d74d3-b7bd-491e-84db-07c449ae1946\") " Nov 03 21:02:18.846123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:18.846093 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64d74d3-b7bd-491e-84db-07c449ae1946-plugins-config-volume" pod "c64d74d3-b7bd-491e-84db-07c449ae1946" (UID: "c64d74d3-b7bd-491e-84db-07c449ae1946"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:18.846588 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:18.846562 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c64d74d3-b7bd-491e-84db-07c449ae1946-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:18.848005 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:18.847978 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64d74d3-b7bd-491e-84db-07c449ae1946-kube-api-access-j7rzj" pod "c64d74d3-b7bd-491e-84db-07c449ae1946" (UID: "c64d74d3-b7bd-491e-84db-07c449ae1946"). InnerVolumeSpecName "kube-api-access-j7rzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:18.947589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:18.947561 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7rzj\" (UniqueName: \"kubernetes.io/projected/c64d74d3-b7bd-491e-84db-07c449ae1946-kube-api-access-j7rzj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.220826 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.220788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:19.290648 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.290612 4711 generic.go:356] "Generic (PLEG): container finished" podID="c64d74d3-b7bd-491e-84db-07c449ae1946" containerID="bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa" exitCode=0 Nov 03 21:02:19.290885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.290667 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" Nov 03 21:02:19.290885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.290695 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" event={"ID":"c64d74d3-b7bd-491e-84db-07c449ae1946","Type":"ContainerDied","Data":"bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa"} Nov 03 21:02:19.290885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.290782 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m" event={"ID":"c64d74d3-b7bd-491e-84db-07c449ae1946","Type":"ContainerDied","Data":"58d3bd271c9cc9d7cd6d63eeac8854f061c1a798bd5e7405159f67e2f4602ffb"} Nov 03 21:02:19.290885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.290812 4711 scope.go:122] "RemoveContainer" containerID="bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa" Nov 03 21:02:19.301724 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.301709 4711 scope.go:122] "RemoveContainer" containerID="bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa" Nov 03 21:02:19.302004 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:02:19.301986 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa\": container with ID starting with bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa not found: ID does not exist" containerID="bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa" Nov 03 21:02:19.302056 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.302013 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa"} err="failed to get container status \"bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa\": rpc error: code = NotFound desc = could not find container \"bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa\": container with ID starting with bc5a5472ac37b5d66482f6226ea37c4cf4b01e161821fdb237b21d3be2d5dcfa not found: ID does not exist" Nov 03 21:02:19.321559 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.321535 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m"] Nov 03 21:02:19.326827 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.326804 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-b13a14-6bb746df8d-fps9m"] Nov 03 21:02:19.774792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:19.774761 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:20.127336 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:20.127246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:20.682054 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:20.682022 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64d74d3-b7bd-491e-84db-07c449ae1946" path="/var/lib/kubelet/pods/c64d74d3-b7bd-491e-84db-07c449ae1946/volumes" Nov 03 21:02:20.933475 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:20.933358 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:21.594234 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:21.594200 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:22.086387 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.086356 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7"] Nov 03 21:02:22.087161 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.087130 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" podUID="fcccaf40-5996-4050-bc42-c543a85cb8a6" containerName="maas-api" containerID="cri-o://9859cad44ee55f020f95300c948cb83e72c2cf5b325b7c6867b8445aa90ed892" gracePeriod=30 Nov 03 21:02:22.167694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.167659 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:22.192905 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.192872 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" containerID="cri-o://38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9" gracePeriod=45 Nov 03 21:02:22.217625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.217586 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" containerID="cri-o://15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e" gracePeriod=45 Nov 03 21:02:22.330516 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.329620 4711 generic.go:356] "Generic (PLEG): container finished" podID="fcccaf40-5996-4050-bc42-c543a85cb8a6" containerID="9859cad44ee55f020f95300c948cb83e72c2cf5b325b7c6867b8445aa90ed892" exitCode=0 Nov 03 21:02:22.330516 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.329707 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" event={"ID":"fcccaf40-5996-4050-bc42-c543a85cb8a6","Type":"ContainerDied","Data":"9859cad44ee55f020f95300c948cb83e72c2cf5b325b7c6867b8445aa90ed892"} Nov 03 21:02:22.430830 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.430782 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": dial tcp 10.133.0.62:8000: connect: connection refused" Nov 03 21:02:22.438200 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.438172 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 21:02:22.487719 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.487610 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:22.529920 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.529890 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls\") pod \"fcccaf40-5996-4050-bc42-c543a85cb8a6\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " Nov 03 21:02:22.530160 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.530131 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-metrics-certs\") pod \"fcccaf40-5996-4050-bc42-c543a85cb8a6\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " Nov 03 21:02:22.530309 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.530182 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcccaf40-5996-4050-bc42-c543a85cb8a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaf40-5996-4050-bc42-c543a85cb8a6-openshift-service-ca\") pod \"fcccaf40-5996-4050-bc42-c543a85cb8a6\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " Nov 03 21:02:22.530309 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.530267 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcccaf40-5996-4050-bc42-c543a85cb8a6-kube-api-access-8mbzg\" (UniqueName: \"kubernetes.io/projected/fcccaf40-5996-4050-bc42-c543a85cb8a6-kube-api-access-8mbzg\") pod \"fcccaf40-5996-4050-bc42-c543a85cb8a6\" (UID: \"fcccaf40-5996-4050-bc42-c543a85cb8a6\") " Nov 03 21:02:22.531031 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.530869 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcccaf40-5996-4050-bc42-c543a85cb8a6-openshift-service-ca" pod "fcccaf40-5996-4050-bc42-c543a85cb8a6" (UID: "fcccaf40-5996-4050-bc42-c543a85cb8a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:22.532554 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.532527 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls" pod "fcccaf40-5996-4050-bc42-c543a85cb8a6" (UID: "fcccaf40-5996-4050-bc42-c543a85cb8a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:22.534044 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.534016 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-metrics-certs" pod "fcccaf40-5996-4050-bc42-c543a85cb8a6" (UID: "fcccaf40-5996-4050-bc42-c543a85cb8a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:22.534570 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.534545 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcccaf40-5996-4050-bc42-c543a85cb8a6-kube-api-access-8mbzg" pod "fcccaf40-5996-4050-bc42-c543a85cb8a6" (UID: "fcccaf40-5996-4050-bc42-c543a85cb8a6"). InnerVolumeSpecName "kube-api-access-8mbzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:22.541594 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.541569 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 21:02:22.617997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.617963 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:22.618808 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.618784 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 21:02:22.631649 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.631586 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-model-cache\") pod \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " Nov 03 21:02:22.631773 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.631668 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tls-certs\") pod \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " Nov 03 21:02:22.631773 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.631699 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tmp-dir\") pod \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " Nov 03 21:02:22.631885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.631795 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-dshm\") pod \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " Nov 03 21:02:22.631885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.631834 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-model-cache" pod "33a4ea2c-bb5c-4324-994e-564a7a2761a8" (UID: "33a4ea2c-bb5c-4324-994e-564a7a2761a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.631999 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.631924 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-home\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-home\") pod \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " Nov 03 21:02:22.632053 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632024 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tmp-dir" pod "33a4ea2c-bb5c-4324-994e-564a7a2761a8" (UID: "33a4ea2c-bb5c-4324-994e-564a7a2761a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.632119 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632045 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a4ea2c-bb5c-4324-994e-564a7a2761a8-kube-api-access-pskzh\" (UniqueName: \"kubernetes.io/projected/33a4ea2c-bb5c-4324-994e-564a7a2761a8-kube-api-access-pskzh\") pod \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\" (UID: \"33a4ea2c-bb5c-4324-994e-564a7a2761a8\") " Nov 03 21:02:22.632238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632218 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-home" pod "33a4ea2c-bb5c-4324-994e-564a7a2761a8" (UID: "33a4ea2c-bb5c-4324-994e-564a7a2761a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.632643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632622 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.632739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632647 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcccaf40-5996-4050-bc42-c543a85cb8a6-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.632739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632662 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.632739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632676 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mbzg\" (UniqueName: \"kubernetes.io/projected/fcccaf40-5996-4050-bc42-c543a85cb8a6-kube-api-access-8mbzg\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.632739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632691 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.632739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632709 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcccaf40-5996-4050-bc42-c543a85cb8a6-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.632739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.632722 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.634266 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.634241 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tls-certs" pod "33a4ea2c-bb5c-4324-994e-564a7a2761a8" (UID: "33a4ea2c-bb5c-4324-994e-564a7a2761a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:22.634923 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.634897 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-dshm" pod "33a4ea2c-bb5c-4324-994e-564a7a2761a8" (UID: "33a4ea2c-bb5c-4324-994e-564a7a2761a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.635196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.635167 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a4ea2c-bb5c-4324-994e-564a7a2761a8-kube-api-access-pskzh" pod "33a4ea2c-bb5c-4324-994e-564a7a2761a8" (UID: "33a4ea2c-bb5c-4324-994e-564a7a2761a8"). InnerVolumeSpecName "kube-api-access-pskzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:22.733441 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.733411 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda8d702-b23b-4df0-8540-febd71b7f7fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/fda8d702-b23b-4df0-8540-febd71b7f7fa-tls-certs\") pod \"fda8d702-b23b-4df0-8540-febd71b7f7fa\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " Nov 03 21:02:22.733644 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.733495 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-model-cache\") pod \"fda8d702-b23b-4df0-8540-febd71b7f7fa\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " Nov 03 21:02:22.733644 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.733595 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda8d702-b23b-4df0-8540-febd71b7f7fa-kube-api-access-lvxt9\" (UniqueName: \"kubernetes.io/projected/fda8d702-b23b-4df0-8540-febd71b7f7fa-kube-api-access-lvxt9\") pod \"fda8d702-b23b-4df0-8540-febd71b7f7fa\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " Nov 03 21:02:22.733786 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.733761 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-dshm\") pod \"fda8d702-b23b-4df0-8540-febd71b7f7fa\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " Nov 03 21:02:22.733845 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.733777 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-model-cache" pod "fda8d702-b23b-4df0-8540-febd71b7f7fa" (UID: "fda8d702-b23b-4df0-8540-febd71b7f7fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.733845 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.733835 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-tmp-dir\") pod \"fda8d702-b23b-4df0-8540-febd71b7f7fa\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " Nov 03 21:02:22.733974 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.733893 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-home\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-home\") pod \"fda8d702-b23b-4df0-8540-febd71b7f7fa\" (UID: \"fda8d702-b23b-4df0-8540-febd71b7f7fa\") " Nov 03 21:02:22.734075 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.734050 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-tmp-dir" pod "fda8d702-b23b-4df0-8540-febd71b7f7fa" (UID: "fda8d702-b23b-4df0-8540-febd71b7f7fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.734176 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.734152 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-home" pod "fda8d702-b23b-4df0-8540-febd71b7f7fa" (UID: "fda8d702-b23b-4df0-8540-febd71b7f7fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.734720 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.734695 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pskzh\" (UniqueName: \"kubernetes.io/projected/33a4ea2c-bb5c-4324-994e-564a7a2761a8-kube-api-access-pskzh\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.734851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.734733 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.734851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.734759 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.734851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.734779 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a4ea2c-bb5c-4324-994e-564a7a2761a8-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.734851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.734793 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.734851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.734806 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33a4ea2c-bb5c-4324-994e-564a7a2761a8-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.735713 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.735690 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda8d702-b23b-4df0-8540-febd71b7f7fa-tls-certs" pod "fda8d702-b23b-4df0-8540-febd71b7f7fa" (UID: "fda8d702-b23b-4df0-8540-febd71b7f7fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:22.735816 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.735773 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda8d702-b23b-4df0-8540-febd71b7f7fa-kube-api-access-lvxt9" pod "fda8d702-b23b-4df0-8540-febd71b7f7fa" (UID: "fda8d702-b23b-4df0-8540-febd71b7f7fa"). InnerVolumeSpecName "kube-api-access-lvxt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:22.735862 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.735832 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-dshm" pod "fda8d702-b23b-4df0-8540-febd71b7f7fa" (UID: "fda8d702-b23b-4df0-8540-febd71b7f7fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:22.836047 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.836013 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda8d702-b23b-4df0-8540-febd71b7f7fa-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.836047 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.836043 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvxt9\" (UniqueName: \"kubernetes.io/projected/fda8d702-b23b-4df0-8540-febd71b7f7fa-kube-api-access-lvxt9\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:22.836047 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:22.836057 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda8d702-b23b-4df0-8540-febd71b7f7fa-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:23.322153 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.322118 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:23.356012 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.355604 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" Nov 03 21:02:23.361673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.355603 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7" event={"ID":"fcccaf40-5996-4050-bc42-c543a85cb8a6","Type":"ContainerDied","Data":"73211205f37a1c5c575a4c0deae0991a81a0ca8a03140da64585ebd257bda8c2"} Nov 03 21:02:23.361845 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.361678 4711 scope.go:122] "RemoveContainer" containerID="9859cad44ee55f020f95300c948cb83e72c2cf5b325b7c6867b8445aa90ed892" Nov 03 21:02:23.366917 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.366073 4711 generic.go:356] "Generic (PLEG): container finished" podID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerID="38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9" exitCode=0 Nov 03 21:02:23.366917 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.366256 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" event={"ID":"fda8d702-b23b-4df0-8540-febd71b7f7fa","Type":"ContainerDied","Data":"38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9"} Nov 03 21:02:23.366917 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.366296 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" event={"ID":"fda8d702-b23b-4df0-8540-febd71b7f7fa","Type":"ContainerDied","Data":"14f55123187a63c974bec758253ee38673b23aae12233656048892713401b1b6"} Nov 03 21:02:23.366917 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.366365 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr" Nov 03 21:02:23.368556 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.368529 4711 generic.go:356] "Generic (PLEG): container finished" podID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerID="15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e" exitCode=0 Nov 03 21:02:23.368723 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.368578 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" Nov 03 21:02:23.368723 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.368623 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" event={"ID":"33a4ea2c-bb5c-4324-994e-564a7a2761a8","Type":"ContainerDied","Data":"15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e"} Nov 03 21:02:23.368723 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.368707 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq" event={"ID":"33a4ea2c-bb5c-4324-994e-564a7a2761a8","Type":"ContainerDied","Data":"d64a6c19ca64db4b9d4d21a62065f62364174260b6855f8977f1fe024c4fc92f"} Nov 03 21:02:23.377376 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.377361 4711 scope.go:122] "RemoveContainer" containerID="38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9" Nov 03 21:02:23.385419 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.385395 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7"] Nov 03 21:02:23.387063 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.387045 4711 scope.go:122] "RemoveContainer" containerID="38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9" Nov 03 21:02:23.387357 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:02:23.387339 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9\": container with ID starting with 38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9 not found: ID does not exist" containerID="38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9" Nov 03 21:02:23.387409 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.387365 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9"} err="failed to get container status \"38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9\": rpc error: code = NotFound desc = could not find container \"38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9\": container with ID starting with 38bcd2b0b63abc721eb38a47d1cc21c1c2f79259e960e6231739646838dad1e9 not found: ID does not exist" Nov 03 21:02:23.387409 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.387384 4711 scope.go:122] "RemoveContainer" containerID="15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e" Nov 03 21:02:23.389846 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.389826 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-14ebbb-7b89968f4-fmvx7"] Nov 03 21:02:23.395908 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.395891 4711 scope.go:122] "RemoveContainer" containerID="15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e" Nov 03 21:02:23.396166 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:02:23.396145 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e\": container with ID starting with 15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e not found: ID does not exist" containerID="15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e" Nov 03 21:02:23.396214 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.396174 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e"} err="failed to get container status \"15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e\": rpc error: code = NotFound desc = could not find container \"15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e\": container with ID starting with 15c49d137aa9b8181308cdf7df260ea3e06ec6488ecb3ccc0c3b718cc9c6844e not found: ID does not exist" Nov 03 21:02:23.406775 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.406755 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr"] Nov 03 21:02:23.413956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.413931 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/distinct-w1-b13a14-kserve-77cb8cdb56-j7gkr"] Nov 03 21:02:23.424166 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.424140 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq"] Nov 03 21:02:23.427075 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:23.427054 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b13a14/simulator-w1-b13a14-kserve-55d5879596-kwgfq"] Nov 03 21:02:24.176958 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:24.176914 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:24.680537 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:24.680501 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" path="/var/lib/kubelet/pods/33a4ea2c-bb5c-4324-994e-564a7a2761a8/volumes" Nov 03 21:02:24.681146 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:24.681126 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcccaf40-5996-4050-bc42-c543a85cb8a6" path="/var/lib/kubelet/pods/fcccaf40-5996-4050-bc42-c543a85cb8a6/volumes" Nov 03 21:02:24.681611 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:24.681591 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" path="/var/lib/kubelet/pods/fda8d702-b23b-4df0-8540-febd71b7f7fa/volumes" Nov 03 21:02:24.812423 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:24.812372 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:25.498285 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:25.498230 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:26.219796 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:26.219760 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:26.659285 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:26.659254 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:26.807040 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:26.807007 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:26.975761 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:26.975672 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:27.128902 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:27.128863 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:27.279911 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:27.279877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:27.591995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:27.591902 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:27.869580 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:27.869489 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:28.338438 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:28.338396 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:28.564518 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:28.564423 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:28.806627 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:28.806584 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:29.445059 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:29.444992 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:30.207532 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:30.207497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:30.481026 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:30.480933 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:30.987477 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:30.987419 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:31.113214 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:31.113178 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:31.594568 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:31.594522 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:31.830190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:31.830152 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:32.627886 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:32.627832 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:33.403864 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:33.403823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:34.304606 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:34.304571 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:34.750425 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:34.750332 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:34.894291 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:34.894242 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:35.147108 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:35.147073 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:35.972972 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:35.972932 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:36.114008 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:36.113943 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:36.410065 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:36.410036 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:36.548873 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:36.548838 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:36.714734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:36.714649 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:37.397130 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:37.397075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:38.063288 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:38.063245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:38.632356 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:38.632319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:39.012086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:39.012049 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:39.735266 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:39.735210 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:40.616627 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:40.609308 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:41.331542 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:41.331497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:42.161533 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:42.161497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:42.301829 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:42.301787 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:42.433061 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:42.432969 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:42.804402 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:42.804364 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:43.072223 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:43.072122 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:43.224103 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:43.224069 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:43.732704 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:43.732666 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:43.895342 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:43.895303 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:44.024759 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:44.024719 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:44.248661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:44.248626 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:44.383992 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:44.383903 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:44.512722 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:44.512300 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:45.456495 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:45.456425 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:46.346603 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:46.344299 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:47.439048 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:47.439015 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:47.532580 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:02:47.532542 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:02:47.704105 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:47.704024 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:47.897757 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:47.897708 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:48.261335 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:48.261297 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7"] Nov 03 21:02:48.261625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:48.261597 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" containerID="cri-o://fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11" gracePeriod=30 Nov 03 21:02:48.332376 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:48.332310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:48.621946 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:48.621810 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:49.272005 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:49.271962 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:49.407421 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:49.407386 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:49.616826 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:49.612620 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:50.428188 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:50.428154 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:51.279231 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:51.279196 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:51.469783 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:51.469738 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:52.293109 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:52.293076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:52.962142 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:52.962108 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:53.537664 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.537605 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 21:02:53.593697 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.593656 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:53.607618 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.607575 4711 generic.go:356] "Generic (PLEG): container finished" podID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerID="fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11" exitCode=0 Nov 03 21:02:53.607794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.607638 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" Nov 03 21:02:53.607794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.607658 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" event={"ID":"05ed1ab0-b90d-4e18-a33c-40e7da0ce032","Type":"ContainerDied","Data":"fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11"} Nov 03 21:02:53.607794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.607733 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7" event={"ID":"05ed1ab0-b90d-4e18-a33c-40e7da0ce032","Type":"ContainerDied","Data":"64b1ddb640ced9e31662c8fc9a52b4db6f7ac929a176cc4d3f02cea54186aaba"} Nov 03 21:02:53.607794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.607767 4711 scope.go:122] "RemoveContainer" containerID="fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11" Nov 03 21:02:53.620550 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.620469 4711 scope.go:122] "RemoveContainer" containerID="fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11" Nov 03 21:02:53.620745 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:02:53.620724 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11\": container with ID starting with fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11 not found: ID does not exist" containerID="fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11" Nov 03 21:02:53.620844 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.620760 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11"} err="failed to get container status \"fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11\": rpc error: code = NotFound desc = could not find container \"fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11\": container with ID starting with fc4133b376423852b135260a7ea404a0e564e25e381bb686aa5fa94c5e8eff11 not found: ID does not exist" Nov 03 21:02:53.667592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.666440 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istiod-ca-cert\") pod \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " Nov 03 21:02:53.667592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.666546 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-certs\") pod \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " Nov 03 21:02:53.667592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.666581 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-podinfo\") pod \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " Nov 03 21:02:53.667592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.666677 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-data\") pod \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " Nov 03 21:02:53.667592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.666824 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-socket\") pod \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " Nov 03 21:02:53.667592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.666864 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-token\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-token\") pod \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " Nov 03 21:02:53.667592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.666926 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-credential-socket\") pod \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " Nov 03 21:02:53.667592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.667032 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-envoy\") pod \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " Nov 03 21:02:53.667592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.667060 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-kube-api-access-drlrt\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-kube-api-access-drlrt\") pod \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\" (UID: \"05ed1ab0-b90d-4e18-a33c-40e7da0ce032\") " Nov 03 21:02:53.675277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.671209 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-data" pod "05ed1ab0-b90d-4e18-a33c-40e7da0ce032" (UID: "05ed1ab0-b90d-4e18-a33c-40e7da0ce032"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.675277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.671576 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istiod-ca-cert" pod "05ed1ab0-b90d-4e18-a33c-40e7da0ce032" (UID: "05ed1ab0-b90d-4e18-a33c-40e7da0ce032"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:53.675277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.672351 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-credential-socket" pod "05ed1ab0-b90d-4e18-a33c-40e7da0ce032" (UID: "05ed1ab0-b90d-4e18-a33c-40e7da0ce032"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.675277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.672593 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-certs" pod "05ed1ab0-b90d-4e18-a33c-40e7da0ce032" (UID: "05ed1ab0-b90d-4e18-a33c-40e7da0ce032"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.682834 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.680830 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-podinfo" pod "05ed1ab0-b90d-4e18-a33c-40e7da0ce032" (UID: "05ed1ab0-b90d-4e18-a33c-40e7da0ce032"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:53.682834 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.681032 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-kube-api-access-drlrt" pod "05ed1ab0-b90d-4e18-a33c-40e7da0ce032" (UID: "05ed1ab0-b90d-4e18-a33c-40e7da0ce032"). InnerVolumeSpecName "kube-api-access-drlrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:53.682834 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.681519 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-socket" pod "05ed1ab0-b90d-4e18-a33c-40e7da0ce032" (UID: "05ed1ab0-b90d-4e18-a33c-40e7da0ce032"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.683315 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.682985 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-token" pod "05ed1ab0-b90d-4e18-a33c-40e7da0ce032" (UID: "05ed1ab0-b90d-4e18-a33c-40e7da0ce032"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:53.683315 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.683088 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-envoy" pod "05ed1ab0-b90d-4e18-a33c-40e7da0ce032" (UID: "05ed1ab0-b90d-4e18-a33c-40e7da0ce032"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:53.726920 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.726891 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:53.771304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.771264 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.771304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.771297 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.771304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.771312 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.771673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.771324 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.771673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.771335 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.771673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.771346 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.771673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.771359 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.771673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.771370 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.771673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.771382 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drlrt\" (UniqueName: \"kubernetes.io/projected/05ed1ab0-b90d-4e18-a33c-40e7da0ce032-kube-api-access-drlrt\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:53.933087 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.933058 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7"] Nov 03 21:02:53.937615 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:53.937589 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b13a14-openshift-default-5dbd78dd47-8vhj7"] Nov 03 21:02:54.327668 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:54.327630 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:54.680354 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:54.680265 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" path="/var/lib/kubelet/pods/05ed1ab0-b90d-4e18-a33c-40e7da0ce032/volumes" Nov 03 21:02:54.975072 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:54.974981 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck"] Nov 03 21:02:54.975258 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:54.975213 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" podUID="6826f58b-3c04-4873-a9e7-970211daf43b" containerName="discovery" containerID="cri-o://e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb" gracePeriod=30 Nov 03 21:02:55.132671 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.132638 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:55.277033 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.277007 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 21:02:55.391608 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.391570 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6826f58b-3c04-4873-a9e7-970211daf43b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6826f58b-3c04-4873-a9e7-970211daf43b-local-certs\") pod \"6826f58b-3c04-4873-a9e7-970211daf43b\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " Nov 03 21:02:55.391608 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.391611 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-istio-token\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-istio-token\") pod \"6826f58b-3c04-4873-a9e7-970211daf43b\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " Nov 03 21:02:55.392130 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.391671 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-dns-cert\") pod \"6826f58b-3c04-4873-a9e7-970211daf43b\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " Nov 03 21:02:55.392130 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.391764 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-kubeconfig\") pod \"6826f58b-3c04-4873-a9e7-970211daf43b\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " Nov 03 21:02:55.392130 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.391841 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-cacerts\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-cacerts\") pod \"6826f58b-3c04-4873-a9e7-970211daf43b\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " Nov 03 21:02:55.392130 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.391948 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-ca-configmap\") pod \"6826f58b-3c04-4873-a9e7-970211daf43b\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " Nov 03 21:02:55.392400 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.392370 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-ca-configmap" pod "6826f58b-3c04-4873-a9e7-970211daf43b" (UID: "6826f58b-3c04-4873-a9e7-970211daf43b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:55.392540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.392518 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-kube-api-access-zhl4q\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-kube-api-access-zhl4q\") pod \"6826f58b-3c04-4873-a9e7-970211daf43b\" (UID: \"6826f58b-3c04-4873-a9e7-970211daf43b\") " Nov 03 21:02:55.394073 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.394047 4711 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-ca-configmap\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.395530 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.395482 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6826f58b-3c04-4873-a9e7-970211daf43b-local-certs" pod "6826f58b-3c04-4873-a9e7-970211daf43b" (UID: "6826f58b-3c04-4873-a9e7-970211daf43b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:55.395530 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.395436 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-cacerts" pod "6826f58b-3c04-4873-a9e7-970211daf43b" (UID: "6826f58b-3c04-4873-a9e7-970211daf43b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:55.395688 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.395583 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-kubeconfig" pod "6826f58b-3c04-4873-a9e7-970211daf43b" (UID: "6826f58b-3c04-4873-a9e7-970211daf43b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:55.395688 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.395607 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-dns-cert" pod "6826f58b-3c04-4873-a9e7-970211daf43b" (UID: "6826f58b-3c04-4873-a9e7-970211daf43b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:55.396619 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.396593 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-kube-api-access-zhl4q" pod "6826f58b-3c04-4873-a9e7-970211daf43b" (UID: "6826f58b-3c04-4873-a9e7-970211daf43b"). InnerVolumeSpecName "kube-api-access-zhl4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:55.396776 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.396627 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-istio-token" pod "6826f58b-3c04-4873-a9e7-970211daf43b" (UID: "6826f58b-3c04-4873-a9e7-970211daf43b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:55.495792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.495700 4711 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-cacerts\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.495792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.495755 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhl4q\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-kube-api-access-zhl4q\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.495792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.495766 4711 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6826f58b-3c04-4873-a9e7-970211daf43b-local-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.495792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.495776 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6826f58b-3c04-4873-a9e7-970211daf43b-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.495792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.495785 4711 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-csr-dns-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.495792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.495793 4711 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6826f58b-3c04-4873-a9e7-970211daf43b-istio-kubeconfig\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:55.622743 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.622711 4711 generic.go:356] "Generic (PLEG): container finished" podID="6826f58b-3c04-4873-a9e7-970211daf43b" containerID="e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb" exitCode=0 Nov 03 21:02:55.622946 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.622768 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" Nov 03 21:02:55.622946 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.622792 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" event={"ID":"6826f58b-3c04-4873-a9e7-970211daf43b","Type":"ContainerDied","Data":"e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb"} Nov 03 21:02:55.622946 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.622883 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck" event={"ID":"6826f58b-3c04-4873-a9e7-970211daf43b","Type":"ContainerDied","Data":"33a52d8adc2b83fa3b89065f19a491acc72aa53c850f1bf82f265dd0189f3f3d"} Nov 03 21:02:55.622946 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.622910 4711 scope.go:122] "RemoveContainer" containerID="e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb" Nov 03 21:02:55.637152 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.637131 4711 scope.go:122] "RemoveContainer" containerID="e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb" Nov 03 21:02:55.637441 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:02:55.637420 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb\": container with ID starting with e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb not found: ID does not exist" containerID="e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb" Nov 03 21:02:55.637441 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.637487 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb"} err="failed to get container status \"e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb\": rpc error: code = NotFound desc = could not find container \"e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb\": container with ID starting with e179853fdff2403207e97f28faedff10ce56cbef53ab6d62267ffb7b6a9947cb not found: ID does not exist" Nov 03 21:02:55.647558 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.647528 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck"] Nov 03 21:02:55.652127 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.652098 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kgcck"] Nov 03 21:02:55.823940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:55.823896 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:56.605601 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:56.605566 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:56.680113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:56.680079 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6826f58b-3c04-4873-a9e7-970211daf43b" path="/var/lib/kubelet/pods/6826f58b-3c04-4873-a9e7-970211daf43b/volumes" Nov 03 21:02:57.305960 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:57.305925 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:57.899006 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:57.898967 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc"] Nov 03 21:02:57.899521 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:57.899233 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" podUID="0289e060-4030-4262-8f32-2d4d656e71c0" containerName="maas-api" containerID="cri-o://5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785" gracePeriod=30 Nov 03 21:02:57.983828 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:57.983771 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:58.154527 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.150114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:58.229333 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.229308 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 21:02:58.293520 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.293482 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:58.353942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.353908 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0289e060-4030-4262-8f32-2d4d656e71c0-kube-api-access-gbzn5\" (UniqueName: \"kubernetes.io/projected/0289e060-4030-4262-8f32-2d4d656e71c0-kube-api-access-gbzn5\") pod \"0289e060-4030-4262-8f32-2d4d656e71c0\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " Nov 03 21:02:58.354148 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.354033 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-maas-api-tls\") pod \"0289e060-4030-4262-8f32-2d4d656e71c0\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " Nov 03 21:02:58.354230 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.354164 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-metrics-certs\") pod \"0289e060-4030-4262-8f32-2d4d656e71c0\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " Nov 03 21:02:58.354230 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.354204 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0289e060-4030-4262-8f32-2d4d656e71c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0289e060-4030-4262-8f32-2d4d656e71c0-openshift-service-ca\") pod \"0289e060-4030-4262-8f32-2d4d656e71c0\" (UID: \"0289e060-4030-4262-8f32-2d4d656e71c0\") " Nov 03 21:02:58.354660 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.354631 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0289e060-4030-4262-8f32-2d4d656e71c0-openshift-service-ca" pod "0289e060-4030-4262-8f32-2d4d656e71c0" (UID: "0289e060-4030-4262-8f32-2d4d656e71c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:58.356209 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.356178 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0289e060-4030-4262-8f32-2d4d656e71c0-kube-api-access-gbzn5" pod "0289e060-4030-4262-8f32-2d4d656e71c0" (UID: "0289e060-4030-4262-8f32-2d4d656e71c0"). InnerVolumeSpecName "kube-api-access-gbzn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:58.356304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.356180 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-maas-api-tls" pod "0289e060-4030-4262-8f32-2d4d656e71c0" (UID: "0289e060-4030-4262-8f32-2d4d656e71c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:58.356624 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.356599 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-metrics-certs" pod "0289e060-4030-4262-8f32-2d4d656e71c0" (UID: "0289e060-4030-4262-8f32-2d4d656e71c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:58.455858 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.455773 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:58.455858 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.455802 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0289e060-4030-4262-8f32-2d4d656e71c0-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:58.455858 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.455812 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0289e060-4030-4262-8f32-2d4d656e71c0-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:58.455858 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.455822 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzn5\" (UniqueName: \"kubernetes.io/projected/0289e060-4030-4262-8f32-2d4d656e71c0-kube-api-access-gbzn5\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:58.648123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.648092 4711 generic.go:356] "Generic (PLEG): container finished" podID="0289e060-4030-4262-8f32-2d4d656e71c0" containerID="5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785" exitCode=0 Nov 03 21:02:58.648305 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.648137 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" Nov 03 21:02:58.648305 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.648177 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" event={"ID":"0289e060-4030-4262-8f32-2d4d656e71c0","Type":"ContainerDied","Data":"5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785"} Nov 03 21:02:58.648305 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.648230 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc" event={"ID":"0289e060-4030-4262-8f32-2d4d656e71c0","Type":"ContainerDied","Data":"3d61b5a78a0fbab03cd945269e88b6455e6e4ec9fbe2e07c2461efbae17a4b83"} Nov 03 21:02:58.648305 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.648255 4711 scope.go:122] "RemoveContainer" containerID="5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785" Nov 03 21:02:58.659979 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.659961 4711 scope.go:122] "RemoveContainer" containerID="5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785" Nov 03 21:02:58.660264 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:02:58.660248 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785\": container with ID starting with 5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785 not found: ID does not exist" containerID="5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785" Nov 03 21:02:58.660304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.660273 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785"} err="failed to get container status \"5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785\": rpc error: code = NotFound desc = could not find container \"5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785\": container with ID starting with 5d16dc6b4d0b32797f36d526f7be9f7b3a3022d0bca92d1a2c526eceecf83785 not found: ID does not exist" Nov 03 21:02:58.681022 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.680995 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc"] Nov 03 21:02:58.681022 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:58.681038 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bab2c3-77b4fcc95b-g9pxc"] Nov 03 21:02:59.058686 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:59.058647 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:59.780028 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:59.779987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:02:59.906799 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:02:59.906762 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:00.527296 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:00.527256 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:00.680192 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:00.680157 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0289e060-4030-4262-8f32-2d4d656e71c0" path="/var/lib/kubelet/pods/0289e060-4030-4262-8f32-2d4d656e71c0/volumes" Nov 03 21:03:01.138718 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:01.138679 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:01.268237 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:01.268191 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:01.963124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:01.961381 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:02.697201 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:02.697153 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:03.411823 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:03.411783 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:04.109389 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:04.109347 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:04.291261 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:04.291226 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:04.417557 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:04.417472 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:04.869267 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:04.869215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:04.996885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:04.996824 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:05.180674 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:05.180578 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:05.811842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:05.811782 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:05.942323 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:05.942287 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:06.498800 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:06.498758 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:07.189051 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:07.188982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:08.076589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:08.076553 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:08.817456 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:08.817413 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:09.448293 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:09.448258 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:09.577323 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:09.577288 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:10.164423 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:10.164381 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:10.866979 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:10.866947 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:11.116945 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:11.116906 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp"] Nov 03 21:03:11.129734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:11.129652 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p"] Nov 03 21:03:11.234470 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:11.234400 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:11.385832 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:11.385732 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:11.887302 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:11.887262 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:12.313398 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:12.313361 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:12.617575 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:12.617457 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:13.311972 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:13.311934 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd"] Nov 03 21:03:13.325606 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:13.325575 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t"] Nov 03 21:03:13.420258 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:13.420218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:13.544611 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:13.544575 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:13.825809 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:13.825773 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:14.375301 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.375271 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:14.378212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.378183 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6"] Nov 03 21:03:14.378480 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.378421 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" podUID="ecd9ea94-bb2d-44f1-8feb-c68a800f208d" containerName="maas-api" containerID="cri-o://b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8" gracePeriod=30 Nov 03 21:03:14.506625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.506588 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:14.702638 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.702611 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 21:03:14.725710 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.725682 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:14.761852 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.761817 4711 generic.go:356] "Generic (PLEG): container finished" podID="ecd9ea94-bb2d-44f1-8feb-c68a800f208d" containerID="b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8" exitCode=0 Nov 03 21:03:14.762036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.761868 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" Nov 03 21:03:14.762036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.761908 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" event={"ID":"ecd9ea94-bb2d-44f1-8feb-c68a800f208d","Type":"ContainerDied","Data":"b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8"} Nov 03 21:03:14.762036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.761966 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6" event={"ID":"ecd9ea94-bb2d-44f1-8feb-c68a800f208d","Type":"ContainerDied","Data":"53762e7d6dcf60c09b708555a33887110e7a8dd5ca3aa693427917d0baff74c1"} Nov 03 21:03:14.762036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.761998 4711 scope.go:122] "RemoveContainer" containerID="b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8" Nov 03 21:03:14.771809 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.771789 4711 scope.go:122] "RemoveContainer" containerID="b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8" Nov 03 21:03:14.772110 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:14.772092 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8\": container with ID starting with b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8 not found: ID does not exist" containerID="b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8" Nov 03 21:03:14.772153 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.772121 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8"} err="failed to get container status \"b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8\": rpc error: code = NotFound desc = could not find container \"b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8\": container with ID starting with b441a9643f307bdf91f68bdfe50e4c750a9ca49c78c3bce1799aa8bf9e10f8f8 not found: ID does not exist" Nov 03 21:03:14.784815 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.784793 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-metrics-certs\") pod \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " Nov 03 21:03:14.784915 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.784872 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-kube-api-access-w4wwj\" (UniqueName: \"kubernetes.io/projected/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-kube-api-access-w4wwj\") pod \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " Nov 03 21:03:14.784997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.784978 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls\") pod \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " Nov 03 21:03:14.785103 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.785084 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-openshift-service-ca\") pod \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\" (UID: \"ecd9ea94-bb2d-44f1-8feb-c68a800f208d\") " Nov 03 21:03:14.785476 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.785439 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-openshift-service-ca" pod "ecd9ea94-bb2d-44f1-8feb-c68a800f208d" (UID: "ecd9ea94-bb2d-44f1-8feb-c68a800f208d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:14.785711 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.785691 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:14.786903 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.786883 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-metrics-certs" pod "ecd9ea94-bb2d-44f1-8feb-c68a800f208d" (UID: "ecd9ea94-bb2d-44f1-8feb-c68a800f208d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:14.786991 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.786944 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls" pod "ecd9ea94-bb2d-44f1-8feb-c68a800f208d" (UID: "ecd9ea94-bb2d-44f1-8feb-c68a800f208d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:14.787037 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.786992 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-kube-api-access-w4wwj" pod "ecd9ea94-bb2d-44f1-8feb-c68a800f208d" (UID: "ecd9ea94-bb2d-44f1-8feb-c68a800f208d"). InnerVolumeSpecName "kube-api-access-w4wwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:14.887394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.887360 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:14.887394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.887390 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4wwj\" (UniqueName: \"kubernetes.io/projected/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-kube-api-access-w4wwj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:14.887616 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:14.887401 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd9ea94-bb2d-44f1-8feb-c68a800f208d-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:15.086252 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:15.086215 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6"] Nov 03 21:03:15.090950 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:15.090924 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-6e5139-5d677b558f-d9sd6"] Nov 03 21:03:15.381110 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:15.381018 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:15.511003 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:15.510966 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:15.942032 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:15.941984 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk"] Nov 03 21:03:15.942371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:15.942338 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" podUID="739a3255-da88-41b3-bd81-331cdce0a6be" containerName="maas-api" containerID="cri-o://b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d" gracePeriod=30 Nov 03 21:03:16.016655 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.016613 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:16.117642 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.117552 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" podUID="b0599233-76dd-4eb9-9b5d-94a815dedd36" containerName="payload-processing" containerID="cri-o://d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99" gracePeriod=25 Nov 03 21:03:16.132678 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.132337 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" podUID="40815cdd-d7d7-4471-a12c-e16fdccd9617" containerName="payload-pre-processing" containerID="cri-o://273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449" gracePeriod=25 Nov 03 21:03:16.248713 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.248682 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:16.306904 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.306852 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 21:03:16.411078 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.411049 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-maas-api-tls\") pod \"739a3255-da88-41b3-bd81-331cdce0a6be\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " Nov 03 21:03:16.411538 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.411210 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739a3255-da88-41b3-bd81-331cdce0a6be-kube-api-access-vx4dx\" (UniqueName: \"kubernetes.io/projected/739a3255-da88-41b3-bd81-331cdce0a6be-kube-api-access-vx4dx\") pod \"739a3255-da88-41b3-bd81-331cdce0a6be\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " Nov 03 21:03:16.411538 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.411276 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739a3255-da88-41b3-bd81-331cdce0a6be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/739a3255-da88-41b3-bd81-331cdce0a6be-openshift-service-ca\") pod \"739a3255-da88-41b3-bd81-331cdce0a6be\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " Nov 03 21:03:16.411538 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.411324 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-metrics-certs\") pod \"739a3255-da88-41b3-bd81-331cdce0a6be\" (UID: \"739a3255-da88-41b3-bd81-331cdce0a6be\") " Nov 03 21:03:16.417456 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.412001 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739a3255-da88-41b3-bd81-331cdce0a6be-openshift-service-ca" pod "739a3255-da88-41b3-bd81-331cdce0a6be" (UID: "739a3255-da88-41b3-bd81-331cdce0a6be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:16.417456 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.415008 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739a3255-da88-41b3-bd81-331cdce0a6be-kube-api-access-vx4dx" pod "739a3255-da88-41b3-bd81-331cdce0a6be" (UID: "739a3255-da88-41b3-bd81-331cdce0a6be"). InnerVolumeSpecName "kube-api-access-vx4dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:16.417456 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.415082 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-metrics-certs" pod "739a3255-da88-41b3-bd81-331cdce0a6be" (UID: "739a3255-da88-41b3-bd81-331cdce0a6be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:16.417456 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.415201 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-maas-api-tls" pod "739a3255-da88-41b3-bd81-331cdce0a6be" (UID: "739a3255-da88-41b3-bd81-331cdce0a6be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:16.453992 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.453966 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 21:03:16.465190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.465166 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 21:03:16.513437 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.513399 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0599233-76dd-4eb9-9b5d-94a815dedd36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0599233-76dd-4eb9-9b5d-94a815dedd36-plugins-config-volume\") pod \"b0599233-76dd-4eb9-9b5d-94a815dedd36\" (UID: \"b0599233-76dd-4eb9-9b5d-94a815dedd36\") " Nov 03 21:03:16.513623 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.513562 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0599233-76dd-4eb9-9b5d-94a815dedd36-kube-api-access-hpn8n\" (UniqueName: \"kubernetes.io/projected/b0599233-76dd-4eb9-9b5d-94a815dedd36-kube-api-access-hpn8n\") pod \"b0599233-76dd-4eb9-9b5d-94a815dedd36\" (UID: \"b0599233-76dd-4eb9-9b5d-94a815dedd36\") " Nov 03 21:03:16.513760 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.513737 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0599233-76dd-4eb9-9b5d-94a815dedd36-plugins-config-volume" pod "b0599233-76dd-4eb9-9b5d-94a815dedd36" (UID: "b0599233-76dd-4eb9-9b5d-94a815dedd36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:16.514180 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.514159 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/739a3255-da88-41b3-bd81-331cdce0a6be-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:16.514251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.514187 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:16.514251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.514203 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0599233-76dd-4eb9-9b5d-94a815dedd36-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:16.514251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.514217 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/739a3255-da88-41b3-bd81-331cdce0a6be-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:16.514251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.514230 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx4dx\" (UniqueName: \"kubernetes.io/projected/739a3255-da88-41b3-bd81-331cdce0a6be-kube-api-access-vx4dx\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:16.515755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.515732 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0599233-76dd-4eb9-9b5d-94a815dedd36-kube-api-access-hpn8n" pod "b0599233-76dd-4eb9-9b5d-94a815dedd36" (UID: "b0599233-76dd-4eb9-9b5d-94a815dedd36"). InnerVolumeSpecName "kube-api-access-hpn8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:16.614888 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.614859 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40815cdd-d7d7-4471-a12c-e16fdccd9617-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40815cdd-d7d7-4471-a12c-e16fdccd9617-plugins-config-volume\") pod \"40815cdd-d7d7-4471-a12c-e16fdccd9617\" (UID: \"40815cdd-d7d7-4471-a12c-e16fdccd9617\") " Nov 03 21:03:16.615054 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.615040 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40815cdd-d7d7-4471-a12c-e16fdccd9617-kube-api-access-4bbtg\" (UniqueName: \"kubernetes.io/projected/40815cdd-d7d7-4471-a12c-e16fdccd9617-kube-api-access-4bbtg\") pod \"40815cdd-d7d7-4471-a12c-e16fdccd9617\" (UID: \"40815cdd-d7d7-4471-a12c-e16fdccd9617\") " Nov 03 21:03:16.615221 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.615199 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40815cdd-d7d7-4471-a12c-e16fdccd9617-plugins-config-volume" pod "40815cdd-d7d7-4471-a12c-e16fdccd9617" (UID: "40815cdd-d7d7-4471-a12c-e16fdccd9617"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:16.615434 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.615420 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpn8n\" (UniqueName: \"kubernetes.io/projected/b0599233-76dd-4eb9-9b5d-94a815dedd36-kube-api-access-hpn8n\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:16.615534 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.615436 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40815cdd-d7d7-4471-a12c-e16fdccd9617-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:16.617056 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.617036 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40815cdd-d7d7-4471-a12c-e16fdccd9617-kube-api-access-4bbtg" pod "40815cdd-d7d7-4471-a12c-e16fdccd9617" (UID: "40815cdd-d7d7-4471-a12c-e16fdccd9617"). InnerVolumeSpecName "kube-api-access-4bbtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:16.682486 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.682436 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd9ea94-bb2d-44f1-8feb-c68a800f208d" path="/var/lib/kubelet/pods/ecd9ea94-bb2d-44f1-8feb-c68a800f208d/volumes" Nov 03 21:03:16.717021 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.716965 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbtg\" (UniqueName: \"kubernetes.io/projected/40815cdd-d7d7-4471-a12c-e16fdccd9617-kube-api-access-4bbtg\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:16.777360 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.777333 4711 generic.go:356] "Generic (PLEG): container finished" podID="b0599233-76dd-4eb9-9b5d-94a815dedd36" containerID="d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99" exitCode=0 Nov 03 21:03:16.777531 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.777377 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" Nov 03 21:03:16.777531 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.777411 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" event={"ID":"b0599233-76dd-4eb9-9b5d-94a815dedd36","Type":"ContainerDied","Data":"d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99"} Nov 03 21:03:16.777531 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.777491 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp" event={"ID":"b0599233-76dd-4eb9-9b5d-94a815dedd36","Type":"ContainerDied","Data":"e9155046b5f365ab34ca87f70cfa111906d3abf2861f86ebfead038b006ec49a"} Nov 03 21:03:16.777531 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.777520 4711 scope.go:122] "RemoveContainer" containerID="d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99" Nov 03 21:03:16.778738 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.778719 4711 generic.go:356] "Generic (PLEG): container finished" podID="739a3255-da88-41b3-bd81-331cdce0a6be" containerID="b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d" exitCode=0 Nov 03 21:03:16.778841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.778775 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" Nov 03 21:03:16.778841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.778801 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" event={"ID":"739a3255-da88-41b3-bd81-331cdce0a6be","Type":"ContainerDied","Data":"b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d"} Nov 03 21:03:16.778993 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.778860 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk" event={"ID":"739a3255-da88-41b3-bd81-331cdce0a6be","Type":"ContainerDied","Data":"7fbd7867f6d2067908a43f79fa6314ec731522732a826c7c28d65a23dc2c027f"} Nov 03 21:03:16.780535 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.780511 4711 generic.go:356] "Generic (PLEG): container finished" podID="40815cdd-d7d7-4471-a12c-e16fdccd9617" containerID="273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449" exitCode=0 Nov 03 21:03:16.780660 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.780541 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" Nov 03 21:03:16.780660 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.780580 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" event={"ID":"40815cdd-d7d7-4471-a12c-e16fdccd9617","Type":"ContainerDied","Data":"273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449"} Nov 03 21:03:16.780660 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.780623 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p" event={"ID":"40815cdd-d7d7-4471-a12c-e16fdccd9617","Type":"ContainerDied","Data":"1fef1b76431d844091a5837f7f07b52251f3e15819ba0ab91bbb40e587997d7c"} Nov 03 21:03:16.788114 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.788083 4711 scope.go:122] "RemoveContainer" containerID="d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99" Nov 03 21:03:16.788389 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:16.788370 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99\": container with ID starting with d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99 not found: ID does not exist" containerID="d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99" Nov 03 21:03:16.788439 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.788397 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99"} err="failed to get container status \"d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99\": rpc error: code = NotFound desc = could not find container \"d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99\": container with ID starting with d071d5636e5effd65c1e883ba755873772d9fcc016cc61b673f52aa5a7368e99 not found: ID does not exist" Nov 03 21:03:16.788439 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.788413 4711 scope.go:122] "RemoveContainer" containerID="b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d" Nov 03 21:03:16.797513 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.797495 4711 scope.go:122] "RemoveContainer" containerID="b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d" Nov 03 21:03:16.797789 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:16.797772 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d\": container with ID starting with b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d not found: ID does not exist" containerID="b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d" Nov 03 21:03:16.797875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.797795 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d"} err="failed to get container status \"b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d\": rpc error: code = NotFound desc = could not find container \"b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d\": container with ID starting with b2da29548e982db87bf6a3bbc86a35f61a79c33eb16633d00f07a2ff2aace57d not found: ID does not exist" Nov 03 21:03:16.797875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.797810 4711 scope.go:122] "RemoveContainer" containerID="273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449" Nov 03 21:03:16.807065 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.807044 4711 scope.go:122] "RemoveContainer" containerID="273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449" Nov 03 21:03:16.807358 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:16.807340 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449\": container with ID starting with 273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449 not found: ID does not exist" containerID="273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449" Nov 03 21:03:16.807417 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.807368 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449"} err="failed to get container status \"273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449\": rpc error: code = NotFound desc = could not find container \"273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449\": container with ID starting with 273799de261a49e6e121405ef46fd6033cee97346e98abf18a8d8b25b15f9449 not found: ID does not exist" Nov 03 21:03:16.809621 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.809600 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp"] Nov 03 21:03:16.812165 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.812147 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6e5139-84b6cc5c8-88znp"] Nov 03 21:03:16.823409 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.823384 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p"] Nov 03 21:03:16.829241 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.829219 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6e5139-6c59cfcd75-nt27p"] Nov 03 21:03:16.840034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.840016 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk"] Nov 03 21:03:16.845434 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.845415 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-63fde4-686fd7d8c4-8gthk"] Nov 03 21:03:16.934783 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:16.934751 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:17.533969 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:17.533930 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:03:17.646033 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:17.645992 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:18.313037 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:18.312996 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" podUID="c13f2e6b-98a3-4200-be21-b6b28db4d38e" containerName="payload-processing" containerID="cri-o://c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed" gracePeriod=25 Nov 03 21:03:18.314231 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:18.314200 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:18.326073 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:18.326001 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" podUID="9be03ba9-8f44-4bac-a84d-e91310aca456" containerName="payload-pre-processing" containerID="cri-o://cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942" gracePeriod=25 Nov 03 21:03:18.680607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:18.680529 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40815cdd-d7d7-4471-a12c-e16fdccd9617" path="/var/lib/kubelet/pods/40815cdd-d7d7-4471-a12c-e16fdccd9617/volumes" Nov 03 21:03:18.681004 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:18.680989 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739a3255-da88-41b3-bd81-331cdce0a6be" path="/var/lib/kubelet/pods/739a3255-da88-41b3-bd81-331cdce0a6be/volumes" Nov 03 21:03:18.681369 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:18.681354 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0599233-76dd-4eb9-9b5d-94a815dedd36" path="/var/lib/kubelet/pods/b0599233-76dd-4eb9-9b5d-94a815dedd36/volumes" Nov 03 21:03:19.002405 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:19.002321 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:19.704339 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:19.704305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:19.848071 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:19.848035 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:20.224162 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:20.224120 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:20.497842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:20.497758 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz"] Nov 03 21:03:20.498187 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:20.498160 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerName="istio-proxy" containerID="cri-o://231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e" gracePeriod=30 Nov 03 21:03:20.568824 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:20.568795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:20.704418 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:20.704386 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:21.110873 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:21.110834 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:21.727129 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:21.727093 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:21.867263 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:21.867225 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:22.206630 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:22.206590 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:22.476576 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:22.476500 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j"] Nov 03 21:03:22.476800 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:22.476771 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" containerID="cri-o://bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc" gracePeriod=30 Nov 03 21:03:22.554746 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:22.554706 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:22.723241 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:22.723212 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:23.309307 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:23.309268 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:23.455203 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:23.455171 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:24.103825 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:24.103772 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:24.787282 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:24.787233 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:25.473912 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.473874 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" podUID="9be03ba9-8f44-4bac-a84d-e91310aca456" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:25.474071 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.473937 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" podUID="c13f2e6b-98a3-4200-be21-b6b28db4d38e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:25.597930 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.597871 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:25.599983 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:25.599945 4711 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996362b4_6f32_4168_b8de_2433bf0f5cf4.slice/crio-conmon-231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:03:25.767002 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.766977 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 21:03:25.851159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.851115 4711 generic.go:356] "Generic (PLEG): container finished" podID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerID="231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e" exitCode=0 Nov 03 21:03:25.851576 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.851192 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" Nov 03 21:03:25.851576 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.851197 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" event={"ID":"996362b4-6f32-4168-b8de-2433bf0f5cf4","Type":"ContainerDied","Data":"231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e"} Nov 03 21:03:25.851576 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.851274 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz" event={"ID":"996362b4-6f32-4168-b8de-2433bf0f5cf4","Type":"ContainerDied","Data":"91d9cfa1341fc495ff376fc6a2d5c11abd26082ab0ae6aa8a47ddce9acc12fed"} Nov 03 21:03:25.851576 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.851297 4711 scope.go:122] "RemoveContainer" containerID="231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e" Nov 03 21:03:25.861239 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.861218 4711 scope.go:122] "RemoveContainer" containerID="231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e" Nov 03 21:03:25.861888 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:25.861842 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e\": container with ID starting with 231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e not found: ID does not exist" containerID="231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e" Nov 03 21:03:25.862006 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.861906 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e"} err="failed to get container status \"231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e\": rpc error: code = NotFound desc = could not find container \"231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e\": container with ID starting with 231c9faeed970bff067c1f7f65fbf743287af56cc12c614cf229b034f274e15e not found: ID does not exist" Nov 03 21:03:25.925328 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.925301 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4"] Nov 03 21:03:25.936331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936308 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-certs\") pod \"996362b4-6f32-4168-b8de-2433bf0f5cf4\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " Nov 03 21:03:25.936471 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936416 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-credential-socket\") pod \"996362b4-6f32-4168-b8de-2433bf0f5cf4\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " Nov 03 21:03:25.936471 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936458 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996362b4-6f32-4168-b8de-2433bf0f5cf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996362b4-6f32-4168-b8de-2433bf0f5cf4-istiod-ca-cert\") pod \"996362b4-6f32-4168-b8de-2433bf0f5cf4\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " Nov 03 21:03:25.936601 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936587 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-token\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-token\") pod \"996362b4-6f32-4168-b8de-2433bf0f5cf4\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " Nov 03 21:03:25.936669 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936628 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-certs" pod "996362b4-6f32-4168-b8de-2433bf0f5cf4" (UID: "996362b4-6f32-4168-b8de-2433bf0f5cf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:25.936669 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936663 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-kube-api-access-bgkdj\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-kube-api-access-bgkdj\") pod \"996362b4-6f32-4168-b8de-2433bf0f5cf4\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " Nov 03 21:03:25.936793 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936691 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-credential-socket" pod "996362b4-6f32-4168-b8de-2433bf0f5cf4" (UID: "996362b4-6f32-4168-b8de-2433bf0f5cf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:25.936793 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936706 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-envoy\") pod \"996362b4-6f32-4168-b8de-2433bf0f5cf4\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " Nov 03 21:03:25.936793 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936768 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-socket\") pod \"996362b4-6f32-4168-b8de-2433bf0f5cf4\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " Nov 03 21:03:25.936953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936806 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-podinfo\") pod \"996362b4-6f32-4168-b8de-2433bf0f5cf4\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " Nov 03 21:03:25.936953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936834 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-data\") pod \"996362b4-6f32-4168-b8de-2433bf0f5cf4\" (UID: \"996362b4-6f32-4168-b8de-2433bf0f5cf4\") " Nov 03 21:03:25.936953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.936883 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996362b4-6f32-4168-b8de-2433bf0f5cf4-istiod-ca-cert" pod "996362b4-6f32-4168-b8de-2433bf0f5cf4" (UID: "996362b4-6f32-4168-b8de-2433bf0f5cf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:25.937545 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.937217 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-socket" pod "996362b4-6f32-4168-b8de-2433bf0f5cf4" (UID: "996362b4-6f32-4168-b8de-2433bf0f5cf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:25.937678 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.937654 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-data" pod "996362b4-6f32-4168-b8de-2433bf0f5cf4" (UID: "996362b4-6f32-4168-b8de-2433bf0f5cf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:25.938191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.938170 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:25.938191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.938191 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:25.938362 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.938201 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:25.938362 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.938210 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:25.938362 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.938218 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996362b4-6f32-4168-b8de-2433bf0f5cf4-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:25.939289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.939249 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-envoy" pod "996362b4-6f32-4168-b8de-2433bf0f5cf4" (UID: "996362b4-6f32-4168-b8de-2433bf0f5cf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:25.939421 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.939399 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-token" pod "996362b4-6f32-4168-b8de-2433bf0f5cf4" (UID: "996362b4-6f32-4168-b8de-2433bf0f5cf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:25.939754 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.939729 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-kube-api-access-bgkdj" pod "996362b4-6f32-4168-b8de-2433bf0f5cf4" (UID: "996362b4-6f32-4168-b8de-2433bf0f5cf4"). InnerVolumeSpecName "kube-api-access-bgkdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:25.939949 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.939928 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-podinfo" pod "996362b4-6f32-4168-b8de-2433bf0f5cf4" (UID: "996362b4-6f32-4168-b8de-2433bf0f5cf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:25.942083 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:25.942061 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs"] Nov 03 21:03:26.039755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:26.039723 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:26.039755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:26.039756 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgkdj\" (UniqueName: \"kubernetes.io/projected/996362b4-6f32-4168-b8de-2433bf0f5cf4-kube-api-access-bgkdj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:26.039956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:26.039765 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:26.039956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:26.039775 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996362b4-6f32-4168-b8de-2433bf0f5cf4-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:26.073011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:26.072975 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:26.177927 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:26.177896 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz"] Nov 03 21:03:26.182157 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:26.182132 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6e5139-openshift-default-76fb6699b5-5ktbz"] Nov 03 21:03:26.358859 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:26.358769 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:26.679859 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:26.679770 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" path="/var/lib/kubelet/pods/996362b4-6f32-4168-b8de-2433bf0f5cf4/volumes" Nov 03 21:03:27.038518 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.038484 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:27.164354 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.164319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:27.744116 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.744095 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 21:03:27.757932 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.757906 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-credential-socket\") pod \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " Nov 03 21:03:27.758041 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.757948 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-data\") pod \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " Nov 03 21:03:27.758041 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.757977 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-token\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-token\") pod \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " Nov 03 21:03:27.758121 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758090 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-kube-api-access-fgmsq\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-kube-api-access-fgmsq\") pod \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " Nov 03 21:03:27.758164 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758123 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-certs\") pod \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " Nov 03 21:03:27.758214 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758181 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-envoy\") pod \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " Nov 03 21:03:27.758214 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758199 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-credential-socket" pod "f58b4c01-2f45-4b9b-a76c-c6d31c100337" (UID: "f58b4c01-2f45-4b9b-a76c-c6d31c100337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:27.758392 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758217 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-data" pod "f58b4c01-2f45-4b9b-a76c-c6d31c100337" (UID: "f58b4c01-2f45-4b9b-a76c-c6d31c100337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:27.758392 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758222 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-podinfo\") pod \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " Nov 03 21:03:27.758392 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758299 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-socket\") pod \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " Nov 03 21:03:27.758602 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758405 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istiod-ca-cert\") pod \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\" (UID: \"f58b4c01-2f45-4b9b-a76c-c6d31c100337\") " Nov 03 21:03:27.758602 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758427 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-certs" pod "f58b4c01-2f45-4b9b-a76c-c6d31c100337" (UID: "f58b4c01-2f45-4b9b-a76c-c6d31c100337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:27.758602 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758548 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-socket" pod "f58b4c01-2f45-4b9b-a76c-c6d31c100337" (UID: "f58b4c01-2f45-4b9b-a76c-c6d31c100337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:27.758909 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.758884 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istiod-ca-cert" pod "f58b4c01-2f45-4b9b-a76c-c6d31c100337" (UID: "f58b4c01-2f45-4b9b-a76c-c6d31c100337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:27.759092 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.759073 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:27.759141 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.759101 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:27.759141 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.759117 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:27.759141 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.759134 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:27.759242 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.759150 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:27.760183 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.760164 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-token" pod "f58b4c01-2f45-4b9b-a76c-c6d31c100337" (UID: "f58b4c01-2f45-4b9b-a76c-c6d31c100337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:27.760491 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.760473 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-kube-api-access-fgmsq" pod "f58b4c01-2f45-4b9b-a76c-c6d31c100337" (UID: "f58b4c01-2f45-4b9b-a76c-c6d31c100337"). InnerVolumeSpecName "kube-api-access-fgmsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:27.760647 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.760623 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-podinfo" pod "f58b4c01-2f45-4b9b-a76c-c6d31c100337" (UID: "f58b4c01-2f45-4b9b-a76c-c6d31c100337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:27.760647 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.760635 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-envoy" pod "f58b4c01-2f45-4b9b-a76c-c6d31c100337" (UID: "f58b4c01-2f45-4b9b-a76c-c6d31c100337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:27.860353 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.860323 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:27.860353 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.860351 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgmsq\" (UniqueName: \"kubernetes.io/projected/f58b4c01-2f45-4b9b-a76c-c6d31c100337-kube-api-access-fgmsq\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:27.860353 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.860360 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:27.860607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.860369 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58b4c01-2f45-4b9b-a76c-c6d31c100337-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:27.867091 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.867066 4711 generic.go:356] "Generic (PLEG): container finished" podID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerID="bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc" exitCode=0 Nov 03 21:03:27.867203 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.867122 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" Nov 03 21:03:27.867203 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.867152 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" event={"ID":"f58b4c01-2f45-4b9b-a76c-c6d31c100337","Type":"ContainerDied","Data":"bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc"} Nov 03 21:03:27.867294 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.867207 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j" event={"ID":"f58b4c01-2f45-4b9b-a76c-c6d31c100337","Type":"ContainerDied","Data":"cb6a201a066f3902a9c65f814a77601e72e28eae3847f9610e2bd08c63a3a8a9"} Nov 03 21:03:27.867294 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.867230 4711 scope.go:122] "RemoveContainer" containerID="bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc" Nov 03 21:03:27.877678 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.877659 4711 scope.go:122] "RemoveContainer" containerID="8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd" Nov 03 21:03:27.886280 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.886257 4711 scope.go:122] "RemoveContainer" containerID="bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc" Nov 03 21:03:27.886691 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:27.886674 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc\": container with ID starting with bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc not found: ID does not exist" containerID="bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc" Nov 03 21:03:27.886760 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.886700 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc"} err="failed to get container status \"bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc\": rpc error: code = NotFound desc = could not find container \"bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc\": container with ID starting with bc965ed951e558ac0dd2642baab151cf28a46a6c0181a849082e119736cdb3bc not found: ID does not exist" Nov 03 21:03:27.886760 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.886717 4711 scope.go:122] "RemoveContainer" containerID="8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd" Nov 03 21:03:27.887006 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:27.886967 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd\": container with ID starting with 8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd not found: ID does not exist" containerID="8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd" Nov 03 21:03:27.887110 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.887001 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd"} err="failed to get container status \"8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd\": rpc error: code = NotFound desc = could not find container \"8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd\": container with ID starting with 8d154e076ff6e9c57863cb65c10545cab688ce3593d3a9198cfbd2b06a9b16bd not found: ID does not exist" Nov 03 21:03:27.893841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.893818 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j"] Nov 03 21:03:27.898179 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.898159 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-63fde4-openshift-default-7765d5df47-c2b6j"] Nov 03 21:03:27.932190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:27.932163 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:28.680517 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:28.680484 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" path="/var/lib/kubelet/pods/f58b4c01-2f45-4b9b-a76c-c6d31c100337/volumes" Nov 03 21:03:28.694571 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:28.694528 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:29.291552 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:29.291510 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:29.438684 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:29.438647 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:30.227794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:30.227755 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:30.524251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:30.524221 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:30.897288 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:30.897204 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:31.097494 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:31.097457 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:31.592462 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:31.592397 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:32.317632 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:32.317585 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:33.099198 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:33.099161 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:33.310540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:33.310503 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf"] Nov 03 21:03:33.403549 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:33.403414 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:33.633975 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:33.633938 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:34.004349 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:34.004312 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:34.132179 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:34.132142 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:34.707551 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:34.707507 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:35.177157 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:35.177127 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:35.473809 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:35.473719 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" podUID="9be03ba9-8f44-4bac-a84d-e91310aca456" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:35.473809 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:35.473776 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" podUID="c13f2e6b-98a3-4200-be21-b6b28db4d38e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:35.842849 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:35.842807 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:36.502713 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:36.502666 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:36.686703 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:36.686664 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:36.823818 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:36.823783 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:37.246107 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:37.246025 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:37.383935 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:37.383888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:37.546805 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:37.546771 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx"] Nov 03 21:03:37.547327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:37.547024 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" containerID="cri-o://bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231" gracePeriod=30 Nov 03 21:03:37.609278 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:37.609238 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:38.096018 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.095978 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:38.311483 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.311419 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" podUID="7d2b99c0-6966-4d2e-aa40-19afba1be993" containerName="payload-pre-processing" containerID="cri-o://f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b" gracePeriod=25 Nov 03 21:03:38.421509 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.421397 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:38.575792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.575770 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 21:03:38.592913 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.592887 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 21:03:38.657660 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.657632 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 21:03:38.693486 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.693457 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9be03ba9-8f44-4bac-a84d-e91310aca456-kube-api-access-cp92z\" (UniqueName: \"kubernetes.io/projected/9be03ba9-8f44-4bac-a84d-e91310aca456-kube-api-access-cp92z\") pod \"9be03ba9-8f44-4bac-a84d-e91310aca456\" (UID: \"9be03ba9-8f44-4bac-a84d-e91310aca456\") " Nov 03 21:03:38.693663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.693501 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2b99c0-6966-4d2e-aa40-19afba1be993-kube-api-access-q25h9\" (UniqueName: \"kubernetes.io/projected/7d2b99c0-6966-4d2e-aa40-19afba1be993-kube-api-access-q25h9\") pod \"7d2b99c0-6966-4d2e-aa40-19afba1be993\" (UID: \"7d2b99c0-6966-4d2e-aa40-19afba1be993\") " Nov 03 21:03:38.693663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.693530 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9be03ba9-8f44-4bac-a84d-e91310aca456-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be03ba9-8f44-4bac-a84d-e91310aca456-plugins-config-volume\") pod \"9be03ba9-8f44-4bac-a84d-e91310aca456\" (UID: \"9be03ba9-8f44-4bac-a84d-e91310aca456\") " Nov 03 21:03:38.693663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.693569 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2b99c0-6966-4d2e-aa40-19afba1be993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2b99c0-6966-4d2e-aa40-19afba1be993-plugins-config-volume\") pod \"7d2b99c0-6966-4d2e-aa40-19afba1be993\" (UID: \"7d2b99c0-6966-4d2e-aa40-19afba1be993\") " Nov 03 21:03:38.694107 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.694075 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2b99c0-6966-4d2e-aa40-19afba1be993-plugins-config-volume" pod "7d2b99c0-6966-4d2e-aa40-19afba1be993" (UID: "7d2b99c0-6966-4d2e-aa40-19afba1be993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:38.694220 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.694168 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be03ba9-8f44-4bac-a84d-e91310aca456-plugins-config-volume" pod "9be03ba9-8f44-4bac-a84d-e91310aca456" (UID: "9be03ba9-8f44-4bac-a84d-e91310aca456"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:38.695693 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.695668 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be03ba9-8f44-4bac-a84d-e91310aca456-kube-api-access-cp92z" pod "9be03ba9-8f44-4bac-a84d-e91310aca456" (UID: "9be03ba9-8f44-4bac-a84d-e91310aca456"). InnerVolumeSpecName "kube-api-access-cp92z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:38.696199 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.696175 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2b99c0-6966-4d2e-aa40-19afba1be993-kube-api-access-q25h9" pod "7d2b99c0-6966-4d2e-aa40-19afba1be993" (UID: "7d2b99c0-6966-4d2e-aa40-19afba1be993"). InnerVolumeSpecName "kube-api-access-q25h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:38.795386 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.795355 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13f2e6b-98a3-4200-be21-b6b28db4d38e-kube-api-access-kprcd\" (UniqueName: \"kubernetes.io/projected/c13f2e6b-98a3-4200-be21-b6b28db4d38e-kube-api-access-kprcd\") pod \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\" (UID: \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\") " Nov 03 21:03:38.795617 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.795559 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13f2e6b-98a3-4200-be21-b6b28db4d38e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13f2e6b-98a3-4200-be21-b6b28db4d38e-plugins-config-volume\") pod \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\" (UID: \"c13f2e6b-98a3-4200-be21-b6b28db4d38e\") " Nov 03 21:03:38.795905 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.795889 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp92z\" (UniqueName: \"kubernetes.io/projected/9be03ba9-8f44-4bac-a84d-e91310aca456-kube-api-access-cp92z\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:38.795970 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.795907 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q25h9\" (UniqueName: \"kubernetes.io/projected/7d2b99c0-6966-4d2e-aa40-19afba1be993-kube-api-access-q25h9\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:38.795970 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.795916 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9be03ba9-8f44-4bac-a84d-e91310aca456-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:38.795970 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.795927 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2b99c0-6966-4d2e-aa40-19afba1be993-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:38.795970 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.795891 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13f2e6b-98a3-4200-be21-b6b28db4d38e-plugins-config-volume" pod "c13f2e6b-98a3-4200-be21-b6b28db4d38e" (UID: "c13f2e6b-98a3-4200-be21-b6b28db4d38e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:38.797424 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.797390 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13f2e6b-98a3-4200-be21-b6b28db4d38e-kube-api-access-kprcd" pod "c13f2e6b-98a3-4200-be21-b6b28db4d38e" (UID: "c13f2e6b-98a3-4200-be21-b6b28db4d38e"). InnerVolumeSpecName "kube-api-access-kprcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:38.875722 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.875684 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:38.897116 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.897078 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kprcd\" (UniqueName: \"kubernetes.io/projected/c13f2e6b-98a3-4200-be21-b6b28db4d38e-kube-api-access-kprcd\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:38.897116 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.897109 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13f2e6b-98a3-4200-be21-b6b28db4d38e-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:38.938097 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.938024 4711 generic.go:356] "Generic (PLEG): container finished" podID="7d2b99c0-6966-4d2e-aa40-19afba1be993" containerID="f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b" exitCode=0 Nov 03 21:03:38.938097 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.938088 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" Nov 03 21:03:38.938290 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.938103 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" event={"ID":"7d2b99c0-6966-4d2e-aa40-19afba1be993","Type":"ContainerDied","Data":"f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b"} Nov 03 21:03:38.938290 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.938161 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf" event={"ID":"7d2b99c0-6966-4d2e-aa40-19afba1be993","Type":"ContainerDied","Data":"0ad7d54c87656a37daea266ab0f546cc519536ea36c704e65f3407e9b3d65703"} Nov 03 21:03:38.938290 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.938182 4711 scope.go:122] "RemoveContainer" containerID="f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b" Nov 03 21:03:38.939981 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.939957 4711 generic.go:356] "Generic (PLEG): container finished" podID="9be03ba9-8f44-4bac-a84d-e91310aca456" containerID="cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942" exitCode=0 Nov 03 21:03:38.940103 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.940007 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" Nov 03 21:03:38.940103 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.940055 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" event={"ID":"9be03ba9-8f44-4bac-a84d-e91310aca456","Type":"ContainerDied","Data":"cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942"} Nov 03 21:03:38.940103 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.940096 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t" event={"ID":"9be03ba9-8f44-4bac-a84d-e91310aca456","Type":"ContainerDied","Data":"5af753752ae42f7b339b02e5cb78fcecfadddb26d40575286155638236d0493a"} Nov 03 21:03:38.941661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.941622 4711 generic.go:356] "Generic (PLEG): container finished" podID="c13f2e6b-98a3-4200-be21-b6b28db4d38e" containerID="c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed" exitCode=0 Nov 03 21:03:38.941788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.941699 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" event={"ID":"c13f2e6b-98a3-4200-be21-b6b28db4d38e","Type":"ContainerDied","Data":"c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed"} Nov 03 21:03:38.941788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.941741 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" event={"ID":"c13f2e6b-98a3-4200-be21-b6b28db4d38e","Type":"ContainerDied","Data":"f732dca2ac44e45ba572fc274b0ce72bedaf4f548042fff0d544fe68380751af"} Nov 03 21:03:38.941788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.941707 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd" Nov 03 21:03:38.948834 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.948814 4711 scope.go:122] "RemoveContainer" containerID="f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b" Nov 03 21:03:38.949085 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:38.949059 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b\": container with ID starting with f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b not found: ID does not exist" containerID="f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b" Nov 03 21:03:38.949174 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.949091 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b"} err="failed to get container status \"f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b\": rpc error: code = NotFound desc = could not find container \"f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b\": container with ID starting with f270b40776a90fce3373130db4b30ae0bcdbdeb83ba2e20e323039ccff32214b not found: ID does not exist" Nov 03 21:03:38.949174 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.949107 4711 scope.go:122] "RemoveContainer" containerID="cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942" Nov 03 21:03:38.958915 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.958897 4711 scope.go:122] "RemoveContainer" containerID="cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942" Nov 03 21:03:38.959196 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:38.959177 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942\": container with ID starting with cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942 not found: ID does not exist" containerID="cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942" Nov 03 21:03:38.959319 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.959205 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942"} err="failed to get container status \"cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942\": rpc error: code = NotFound desc = could not find container \"cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942\": container with ID starting with cfbf4da3f0ee784015ccdf26c7b8738365bd88de7660c9d970aa5b1da3434942 not found: ID does not exist" Nov 03 21:03:38.959319 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.959226 4711 scope.go:122] "RemoveContainer" containerID="c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed" Nov 03 21:03:38.965776 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.965737 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf"] Nov 03 21:03:38.968185 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.968168 4711 scope.go:122] "RemoveContainer" containerID="c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed" Nov 03 21:03:38.968486 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:38.968466 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed\": container with ID starting with c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed not found: ID does not exist" containerID="c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed" Nov 03 21:03:38.968576 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.968491 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed"} err="failed to get container status \"c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed\": rpc error: code = NotFound desc = could not find container \"c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed\": container with ID starting with c9841fce3f9f31ddb9251a33f944e4b07475b7cfd7b0dee3d1c49c3d688773ed not found: ID does not exist" Nov 03 21:03:38.970492 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.970472 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4f93d1-794f67f74c-f5gbf"] Nov 03 21:03:38.984553 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.984532 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t"] Nov 03 21:03:38.989196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.989177 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-63fde4-7cf975855f-z7z9t"] Nov 03 21:03:39.000000 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:38.999980 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd"] Nov 03 21:03:39.003998 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:39.003977 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-63fde4-cc696dd64-rfnsd"] Nov 03 21:03:39.502317 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:39.502284 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:40.253621 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:40.253586 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:40.679799 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:40.679767 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2b99c0-6966-4d2e-aa40-19afba1be993" path="/var/lib/kubelet/pods/7d2b99c0-6966-4d2e-aa40-19afba1be993/volumes" Nov 03 21:03:40.680183 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:40.680170 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9be03ba9-8f44-4bac-a84d-e91310aca456" path="/var/lib/kubelet/pods/9be03ba9-8f44-4bac-a84d-e91310aca456/volumes" Nov 03 21:03:40.680553 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:40.680538 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13f2e6b-98a3-4200-be21-b6b28db4d38e" path="/var/lib/kubelet/pods/c13f2e6b-98a3-4200-be21-b6b28db4d38e/volumes" Nov 03 21:03:40.921026 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:40.920987 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:40.948689 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:40.948595 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" containerID="cri-o://fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7" gracePeriod=45 Nov 03 21:03:40.963361 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:40.963326 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" containerID="cri-o://de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c" gracePeriod=45 Nov 03 21:03:41.053787 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.053754 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": dial tcp 10.133.0.74:8000: connect: connection refused" Nov 03 21:03:41.085883 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.085843 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused" Nov 03 21:03:41.233066 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.233000 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:41.236662 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.236637 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 21:03:41.258589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.258565 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 21:03:41.327034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.327000 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-dshm\") pod \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " Nov 03 21:03:41.327241 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.327061 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tls-certs\" (UniqueName: \"kubernetes.io/secret/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tls-certs\") pod \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " Nov 03 21:03:41.327241 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.327092 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-model-cache\") pod \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " Nov 03 21:03:41.327241 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.327155 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tmp-dir\") pod \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " Nov 03 21:03:41.327241 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.327224 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-kube-api-access-t5nsz\" (UniqueName: \"kubernetes.io/projected/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-kube-api-access-t5nsz\") pod \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " Nov 03 21:03:41.327547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.327349 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-home\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-home\") pod \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\" (UID: \"55ad5f3f-8ae3-4c11-804f-2eba281ec11d\") " Nov 03 21:03:41.327547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.327421 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-model-cache" pod "55ad5f3f-8ae3-4c11-804f-2eba281ec11d" (UID: "55ad5f3f-8ae3-4c11-804f-2eba281ec11d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.327547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.327482 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tmp-dir" pod "55ad5f3f-8ae3-4c11-804f-2eba281ec11d" (UID: "55ad5f3f-8ae3-4c11-804f-2eba281ec11d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.327773 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.327702 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-home" pod "55ad5f3f-8ae3-4c11-804f-2eba281ec11d" (UID: "55ad5f3f-8ae3-4c11-804f-2eba281ec11d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.328146 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.328117 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.328270 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.328149 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.328270 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.328164 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.329486 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.329436 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tls-certs" pod "55ad5f3f-8ae3-4c11-804f-2eba281ec11d" (UID: "55ad5f3f-8ae3-4c11-804f-2eba281ec11d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:41.329755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.329734 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-dshm" pod "55ad5f3f-8ae3-4c11-804f-2eba281ec11d" (UID: "55ad5f3f-8ae3-4c11-804f-2eba281ec11d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.330521 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.329909 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-kube-api-access-t5nsz" pod "55ad5f3f-8ae3-4c11-804f-2eba281ec11d" (UID: "55ad5f3f-8ae3-4c11-804f-2eba281ec11d"). InnerVolumeSpecName "kube-api-access-t5nsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:41.429096 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429060 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-kube-api-access-n87ng\" (UniqueName: \"kubernetes.io/projected/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-kube-api-access-n87ng\") pod \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " Nov 03 21:03:41.429268 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429152 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-model-cache\") pod \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " Nov 03 21:03:41.429268 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429182 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tmp-dir\") pod \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " Nov 03 21:03:41.429268 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429252 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tls-certs\") pod \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " Nov 03 21:03:41.429391 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429320 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-home\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-home\") pod \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " Nov 03 21:03:41.429391 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429367 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-dshm\") pod \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\" (UID: \"4b9d0cb1-0a0f-497d-a6a2-483f8338466b\") " Nov 03 21:03:41.429539 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429474 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-model-cache" pod "4b9d0cb1-0a0f-497d-a6a2-483f8338466b" (UID: "4b9d0cb1-0a0f-497d-a6a2-483f8338466b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.429539 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429524 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tmp-dir" pod "4b9d0cb1-0a0f-497d-a6a2-483f8338466b" (UID: "4b9d0cb1-0a0f-497d-a6a2-483f8338466b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.429674 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429652 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-home" pod "4b9d0cb1-0a0f-497d-a6a2-483f8338466b" (UID: "4b9d0cb1-0a0f-497d-a6a2-483f8338466b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.429923 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429903 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.430011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429927 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.430011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429942 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.430011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429955 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5nsz\" (UniqueName: \"kubernetes.io/projected/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-kube-api-access-t5nsz\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.430011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429972 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.430011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.429986 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ad5f3f-8ae3-4c11-804f-2eba281ec11d-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.431486 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.431460 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-kube-api-access-n87ng" pod "4b9d0cb1-0a0f-497d-a6a2-483f8338466b" (UID: "4b9d0cb1-0a0f-497d-a6a2-483f8338466b"). InnerVolumeSpecName "kube-api-access-n87ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:41.431588 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.431490 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-dshm" pod "4b9d0cb1-0a0f-497d-a6a2-483f8338466b" (UID: "4b9d0cb1-0a0f-497d-a6a2-483f8338466b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:41.431588 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.431507 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tls-certs" pod "4b9d0cb1-0a0f-497d-a6a2-483f8338466b" (UID: "4b9d0cb1-0a0f-497d-a6a2-483f8338466b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:41.531207 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.531174 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.531207 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.531203 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.531410 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.531213 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n87ng\" (UniqueName: \"kubernetes.io/projected/4b9d0cb1-0a0f-497d-a6a2-483f8338466b-kube-api-access-n87ng\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:41.624481 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.624431 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:41.965411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.965016 4711 generic.go:356] "Generic (PLEG): container finished" podID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerID="fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7" exitCode=0 Nov 03 21:03:41.965411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.965064 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" Nov 03 21:03:41.965411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.965101 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" event={"ID":"55ad5f3f-8ae3-4c11-804f-2eba281ec11d","Type":"ContainerDied","Data":"fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7"} Nov 03 21:03:41.965411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.965148 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4" event={"ID":"55ad5f3f-8ae3-4c11-804f-2eba281ec11d","Type":"ContainerDied","Data":"ec7541bf82290ae8976d7c6a286b8ba313e8d68c0ef39de8945648f3400637c4"} Nov 03 21:03:41.965411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.965172 4711 scope.go:122] "RemoveContainer" containerID="fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7" Nov 03 21:03:41.967113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.967070 4711 generic.go:356] "Generic (PLEG): container finished" podID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerID="de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c" exitCode=0 Nov 03 21:03:41.967314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.967205 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" event={"ID":"4b9d0cb1-0a0f-497d-a6a2-483f8338466b","Type":"ContainerDied","Data":"de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c"} Nov 03 21:03:41.967314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.967235 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" event={"ID":"4b9d0cb1-0a0f-497d-a6a2-483f8338466b","Type":"ContainerDied","Data":"870f11e291db86d444cf490d638b4cf3cbb8c226d03b8fabded2be45b3c7204c"} Nov 03 21:03:41.967314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.967297 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs" Nov 03 21:03:41.976009 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.975993 4711 scope.go:122] "RemoveContainer" containerID="fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7" Nov 03 21:03:41.976322 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:41.976243 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7\": container with ID starting with fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7 not found: ID does not exist" containerID="fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7" Nov 03 21:03:41.976322 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.976267 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7"} err="failed to get container status \"fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7\": rpc error: code = NotFound desc = could not find container \"fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7\": container with ID starting with fc59b1b365266e3ffc77ca48fd1d62e35aa2d008ee5ab62436cee66cf22fa7d7 not found: ID does not exist" Nov 03 21:03:41.976322 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.976283 4711 scope.go:122] "RemoveContainer" containerID="de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c" Nov 03 21:03:41.985795 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.985776 4711 scope.go:122] "RemoveContainer" containerID="de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c" Nov 03 21:03:41.986081 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:41.986064 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c\": container with ID starting with de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c not found: ID does not exist" containerID="de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c" Nov 03 21:03:41.986153 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.986088 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c"} err="failed to get container status \"de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c\": rpc error: code = NotFound desc = could not find container \"de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c\": container with ID starting with de798e7c21e1a91e1ba7841bf1c68fd5a515254a2b24a2c3b3a022674244685c not found: ID does not exist" Nov 03 21:03:41.992635 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.992615 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs"] Nov 03 21:03:41.998355 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:41.998330 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/premium-w3-4f93d1-kserve-5f78dcb89b-gt4xs"] Nov 03 21:03:42.009788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.009765 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4"] Nov 03 21:03:42.014614 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.014594 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4f93d1/distinct-2-w3-4f93d1-kserve-79bd577476-qm5t4"] Nov 03 21:03:42.524086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.524049 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:42.681373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.681337 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" path="/var/lib/kubelet/pods/4b9d0cb1-0a0f-497d-a6a2-483f8338466b/volumes" Nov 03 21:03:42.681828 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.681812 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" path="/var/lib/kubelet/pods/55ad5f3f-8ae3-4c11-804f-2eba281ec11d/volumes" Nov 03 21:03:42.817788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.817763 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 21:03:42.948714 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.948677 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159c4fe1-8080-4f38-b1d9-d67250329c9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159c4fe1-8080-4f38-b1d9-d67250329c9a-istiod-ca-cert\") pod \"159c4fe1-8080-4f38-b1d9-d67250329c9a\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " Nov 03 21:03:42.948905 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.948747 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-envoy\") pod \"159c4fe1-8080-4f38-b1d9-d67250329c9a\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " Nov 03 21:03:42.948905 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.948831 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-kube-api-access-xtq5z\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-kube-api-access-xtq5z\") pod \"159c4fe1-8080-4f38-b1d9-d67250329c9a\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " Nov 03 21:03:42.948905 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.948888 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-socket\") pod \"159c4fe1-8080-4f38-b1d9-d67250329c9a\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " Nov 03 21:03:42.949083 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.948920 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-certs\") pod \"159c4fe1-8080-4f38-b1d9-d67250329c9a\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " Nov 03 21:03:42.949083 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.948984 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-podinfo\") pod \"159c4fe1-8080-4f38-b1d9-d67250329c9a\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " Nov 03 21:03:42.949083 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949026 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-data\") pod \"159c4fe1-8080-4f38-b1d9-d67250329c9a\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " Nov 03 21:03:42.949236 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949100 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159c4fe1-8080-4f38-b1d9-d67250329c9a-istiod-ca-cert" pod "159c4fe1-8080-4f38-b1d9-d67250329c9a" (UID: "159c4fe1-8080-4f38-b1d9-d67250329c9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:42.949236 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949150 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-socket" pod "159c4fe1-8080-4f38-b1d9-d67250329c9a" (UID: "159c4fe1-8080-4f38-b1d9-d67250329c9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:42.949358 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949135 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-credential-socket\") pod \"159c4fe1-8080-4f38-b1d9-d67250329c9a\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " Nov 03 21:03:42.949358 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949333 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-token\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-token\") pod \"159c4fe1-8080-4f38-b1d9-d67250329c9a\" (UID: \"159c4fe1-8080-4f38-b1d9-d67250329c9a\") " Nov 03 21:03:42.949490 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949366 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-certs" pod "159c4fe1-8080-4f38-b1d9-d67250329c9a" (UID: "159c4fe1-8080-4f38-b1d9-d67250329c9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:42.949490 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949396 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-credential-socket" pod "159c4fe1-8080-4f38-b1d9-d67250329c9a" (UID: "159c4fe1-8080-4f38-b1d9-d67250329c9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:42.949490 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949403 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-data" pod "159c4fe1-8080-4f38-b1d9-d67250329c9a" (UID: "159c4fe1-8080-4f38-b1d9-d67250329c9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:42.949844 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949820 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159c4fe1-8080-4f38-b1d9-d67250329c9a-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:42.949940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949848 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:42.949940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949861 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:42.949940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949874 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:42.949940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.949882 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:42.951353 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.951325 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-token" pod "159c4fe1-8080-4f38-b1d9-d67250329c9a" (UID: "159c4fe1-8080-4f38-b1d9-d67250329c9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:42.951558 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.951535 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-podinfo" pod "159c4fe1-8080-4f38-b1d9-d67250329c9a" (UID: "159c4fe1-8080-4f38-b1d9-d67250329c9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:42.951648 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.951538 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-envoy" pod "159c4fe1-8080-4f38-b1d9-d67250329c9a" (UID: "159c4fe1-8080-4f38-b1d9-d67250329c9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:42.951648 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.951586 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-kube-api-access-xtq5z" pod "159c4fe1-8080-4f38-b1d9-d67250329c9a" (UID: "159c4fe1-8080-4f38-b1d9-d67250329c9a"). InnerVolumeSpecName "kube-api-access-xtq5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:42.977220 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.977194 4711 generic.go:356] "Generic (PLEG): container finished" podID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerID="bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231" exitCode=0 Nov 03 21:03:42.977344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.977246 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" Nov 03 21:03:42.977344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.977269 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" event={"ID":"159c4fe1-8080-4f38-b1d9-d67250329c9a","Type":"ContainerDied","Data":"bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231"} Nov 03 21:03:42.977344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.977318 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx" event={"ID":"159c4fe1-8080-4f38-b1d9-d67250329c9a","Type":"ContainerDied","Data":"b3566f42d236981b1ec14a6aaa9b4cfeadaa54aa5e30f44b0aa454a95a00d6bc"} Nov 03 21:03:42.977344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.977342 4711 scope.go:122] "RemoveContainer" containerID="bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231" Nov 03 21:03:42.988915 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.988899 4711 scope.go:122] "RemoveContainer" containerID="8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02" Nov 03 21:03:42.997408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.997387 4711 scope.go:122] "RemoveContainer" containerID="bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231" Nov 03 21:03:42.997722 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:42.997697 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231\": container with ID starting with bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231 not found: ID does not exist" containerID="bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231" Nov 03 21:03:42.997819 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.997732 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231"} err="failed to get container status \"bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231\": rpc error: code = NotFound desc = could not find container \"bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231\": container with ID starting with bfe3b19e31986b781fd6fab65f3b8c27d9a625e15576879cd798a16c64dd0231 not found: ID does not exist" Nov 03 21:03:42.997819 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.997752 4711 scope.go:122] "RemoveContainer" containerID="8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02" Nov 03 21:03:42.998005 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:42.997987 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02\": container with ID starting with 8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02 not found: ID does not exist" containerID="8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02" Nov 03 21:03:42.998054 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:42.998011 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02"} err="failed to get container status \"8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02\": rpc error: code = NotFound desc = could not find container \"8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02\": container with ID starting with 8ec6451be23fd420ca9d1438c478c5e90cf84aac452ec2a4420671a7cf242f02 not found: ID does not exist" Nov 03 21:03:43.005607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:43.005581 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx"] Nov 03 21:03:43.009722 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:43.009702 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3bcc2b-openshift-default-8d8447459-mhhvx"] Nov 03 21:03:43.051522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:43.051462 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:43.051522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:43.051493 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtq5z\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-kube-api-access-xtq5z\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:43.051522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:43.051509 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:43.051670 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:43.051523 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159c4fe1-8080-4f38-b1d9-d67250329c9a-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:43.313395 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:43.313296 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:44.148129 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:44.148086 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:44.680207 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:44.680175 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" path="/var/lib/kubelet/pods/159c4fe1-8080-4f38-b1d9-d67250329c9a/volumes" Nov 03 21:03:44.824853 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:44.824788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:44.952616 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:44.952528 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:45.577327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:45.577294 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:45.710724 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:45.710671 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:46.116874 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.116835 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb"] Nov 03 21:03:46.117211 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.117181 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" podUID="60a690b0-4534-4550-93dd-313405071b7e" containerName="maas-api" containerID="cri-o://4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99" gracePeriod=30 Nov 03 21:03:46.256734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.256696 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:46.384378 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.384351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:46.392567 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.392543 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 21:03:46.489645 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.489615 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-maas-api-tls\") pod \"60a690b0-4534-4550-93dd-313405071b7e\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " Nov 03 21:03:46.489850 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.489661 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a690b0-4534-4550-93dd-313405071b7e-kube-api-access-znhbn\" (UniqueName: \"kubernetes.io/projected/60a690b0-4534-4550-93dd-313405071b7e-kube-api-access-znhbn\") pod \"60a690b0-4534-4550-93dd-313405071b7e\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " Nov 03 21:03:46.489850 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.489713 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a690b0-4534-4550-93dd-313405071b7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a690b0-4534-4550-93dd-313405071b7e-openshift-service-ca\") pod \"60a690b0-4534-4550-93dd-313405071b7e\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " Nov 03 21:03:46.489850 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.489831 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-metrics-certs\") pod \"60a690b0-4534-4550-93dd-313405071b7e\" (UID: \"60a690b0-4534-4550-93dd-313405071b7e\") " Nov 03 21:03:46.490191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.490158 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a690b0-4534-4550-93dd-313405071b7e-openshift-service-ca" pod "60a690b0-4534-4550-93dd-313405071b7e" (UID: "60a690b0-4534-4550-93dd-313405071b7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:46.491754 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.491731 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-maas-api-tls" pod "60a690b0-4534-4550-93dd-313405071b7e" (UID: "60a690b0-4534-4550-93dd-313405071b7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:46.491951 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.491930 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-metrics-certs" pod "60a690b0-4534-4550-93dd-313405071b7e" (UID: "60a690b0-4534-4550-93dd-313405071b7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:46.492043 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.492026 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a690b0-4534-4550-93dd-313405071b7e-kube-api-access-znhbn" pod "60a690b0-4534-4550-93dd-313405071b7e" (UID: "60a690b0-4534-4550-93dd-313405071b7e"). InnerVolumeSpecName "kube-api-access-znhbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:46.512280 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.512252 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:46.591304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.591270 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:46.591304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.591297 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60a690b0-4534-4550-93dd-313405071b7e-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:46.591304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.591308 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znhbn\" (UniqueName: \"kubernetes.io/projected/60a690b0-4534-4550-93dd-313405071b7e-kube-api-access-znhbn\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:46.591787 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:46.591320 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a690b0-4534-4550-93dd-313405071b7e-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.003851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.003771 4711 generic.go:356] "Generic (PLEG): container finished" podID="60a690b0-4534-4550-93dd-313405071b7e" containerID="4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99" exitCode=0 Nov 03 21:03:47.003851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.003828 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" Nov 03 21:03:47.004168 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.003850 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" event={"ID":"60a690b0-4534-4550-93dd-313405071b7e","Type":"ContainerDied","Data":"4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99"} Nov 03 21:03:47.004168 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.003902 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb" event={"ID":"60a690b0-4534-4550-93dd-313405071b7e","Type":"ContainerDied","Data":"28543e9ba4241b4497443884b7ee3253eb25acc109e618db60e30ba8567c9579"} Nov 03 21:03:47.004168 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.003924 4711 scope.go:122] "RemoveContainer" containerID="4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99" Nov 03 21:03:47.013259 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.013238 4711 scope.go:122] "RemoveContainer" containerID="4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99" Nov 03 21:03:47.013534 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:47.013515 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99\": container with ID starting with 4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99 not found: ID does not exist" containerID="4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99" Nov 03 21:03:47.013598 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.013541 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99"} err="failed to get container status \"4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99\": rpc error: code = NotFound desc = could not find container \"4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99\": container with ID starting with 4c83437d4380f0a219dc4abb44243e8c116a61ed19d733764259d00f8e781c99 not found: ID does not exist" Nov 03 21:03:47.022420 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.022399 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb"] Nov 03 21:03:47.026486 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.026468 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-37aa95-75b5f87d4b-8ktrb"] Nov 03 21:03:47.090835 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.090791 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:47.530995 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:47.530952 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:03:47.751191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.751157 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:47.927122 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:47.927036 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:48.424497 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:48.424439 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:48.678836 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:48.678767 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a690b0-4534-4550-93dd-313405071b7e" path="/var/lib/kubelet/pods/60a690b0-4534-4550-93dd-313405071b7e/volumes" Nov 03 21:03:49.098193 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:49.098157 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:49.807388 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:49.807344 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:50.299543 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.299508 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9"] Nov 03 21:03:50.300063 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.299793 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" podUID="0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" containerName="maas-api" containerID="cri-o://07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08" gracePeriod=30 Nov 03 21:03:50.383050 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.383004 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:50.572977 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.572951 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 21:03:50.630902 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.630874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:50.644647 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.644623 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-openshift-service-ca\") pod \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " Nov 03 21:03:50.644782 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.644707 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-maas-api-tls\") pod \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " Nov 03 21:03:50.644782 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.644749 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-metrics-certs\") pod \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " Nov 03 21:03:50.644782 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.644769 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-kube-api-access-hnrzc\" (UniqueName: \"kubernetes.io/projected/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-kube-api-access-hnrzc\") pod \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\" (UID: \"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d\") " Nov 03 21:03:50.645054 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.645026 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-openshift-service-ca" pod "0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" (UID: "0bf9dbbc-4dda-4af1-9dd6-2da60134f11d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:50.646895 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.646869 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-maas-api-tls" pod "0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" (UID: "0bf9dbbc-4dda-4af1-9dd6-2da60134f11d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:50.646895 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.646884 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-metrics-certs" pod "0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" (UID: "0bf9dbbc-4dda-4af1-9dd6-2da60134f11d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:50.647059 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.647035 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-kube-api-access-hnrzc" pod "0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" (UID: "0bf9dbbc-4dda-4af1-9dd6-2da60134f11d"). InnerVolumeSpecName "kube-api-access-hnrzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:50.745999 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.745944 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.745999 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.745981 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.745999 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.745995 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.745999 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:50.746008 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnrzc\" (UniqueName: \"kubernetes.io/projected/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d-kube-api-access-hnrzc\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:51.028732 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.028700 4711 generic.go:356] "Generic (PLEG): container finished" podID="0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" containerID="07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08" exitCode=0 Nov 03 21:03:51.028942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.028759 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" Nov 03 21:03:51.028942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.028769 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" event={"ID":"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d","Type":"ContainerDied","Data":"07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08"} Nov 03 21:03:51.028942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.028885 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9" event={"ID":"0bf9dbbc-4dda-4af1-9dd6-2da60134f11d","Type":"ContainerDied","Data":"98bd98c7c783c8936b952558f7a233afd04dcc5b1213a2f6ce533fbfa3bfcb57"} Nov 03 21:03:51.028942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.028905 4711 scope.go:122] "RemoveContainer" containerID="07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08" Nov 03 21:03:51.038317 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.038303 4711 scope.go:122] "RemoveContainer" containerID="07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08" Nov 03 21:03:51.038699 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:03:51.038673 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08\": container with ID starting with 07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08 not found: ID does not exist" containerID="07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08" Nov 03 21:03:51.038800 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.038703 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08"} err="failed to get container status \"07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08\": rpc error: code = NotFound desc = could not find container \"07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08\": container with ID starting with 07ef26b4abc2545e7c4988da7e2a46025a8ce2b4cb78132cd696ee22efbd3f08 not found: ID does not exist" Nov 03 21:03:51.048801 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.048763 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9"] Nov 03 21:03:51.053232 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.053209 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4f93d1-76dc7c7767-qd5b9"] Nov 03 21:03:51.320877 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:51.320779 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:52.014677 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:52.014644 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:52.520475 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:52.520426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:52.674525 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:52.674482 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:52.679627 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:52.679601 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" path="/var/lib/kubelet/pods/0bf9dbbc-4dda-4af1-9dd6-2da60134f11d/volumes" Nov 03 21:03:52.814573 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:52.814481 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:52.997910 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:52.997867 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:53.326920 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:53.326889 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:53.710960 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:53.710877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:54.128967 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:54.128935 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:54.339325 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:54.339287 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:54.469434 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:54.469351 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:55.165109 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:55.165076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:55.886038 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:55.885990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:56.565111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:56.565071 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:57.225894 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:57.225858 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:57.892470 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:57.892421 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:58.574871 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:58.574837 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:59.487124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:59.487084 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:03:59.928283 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:03:59.928237 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:00.176964 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:00.176929 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:00.311483 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:00.311427 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:00.454472 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:00.454409 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:01.171779 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.171740 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn"] Nov 03 21:04:01.172291 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.171965 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" podUID="4c6a9a9c-bd88-4ce1-a661-691688449924" containerName="maas-api" containerID="cri-o://c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969" gracePeriod=30 Nov 03 21:04:01.200517 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.200485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:01.327787 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.327754 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:01.471288 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.471263 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 21:04:01.568775 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.568740 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-metrics-certs\") pod \"4c6a9a9c-bd88-4ce1-a661-691688449924\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " Nov 03 21:04:01.568935 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.568893 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6a9a9c-bd88-4ce1-a661-691688449924-kube-api-access-l959m\" (UniqueName: \"kubernetes.io/projected/4c6a9a9c-bd88-4ce1-a661-691688449924-kube-api-access-l959m\") pod \"4c6a9a9c-bd88-4ce1-a661-691688449924\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " Nov 03 21:04:01.568976 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.568963 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6a9a9c-bd88-4ce1-a661-691688449924-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6a9a9c-bd88-4ce1-a661-691688449924-openshift-service-ca\") pod \"4c6a9a9c-bd88-4ce1-a661-691688449924\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " Nov 03 21:04:01.569028 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.569003 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-maas-api-tls\") pod \"4c6a9a9c-bd88-4ce1-a661-691688449924\" (UID: \"4c6a9a9c-bd88-4ce1-a661-691688449924\") " Nov 03 21:04:01.569280 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.569257 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6a9a9c-bd88-4ce1-a661-691688449924-openshift-service-ca" pod "4c6a9a9c-bd88-4ce1-a661-691688449924" (UID: "4c6a9a9c-bd88-4ce1-a661-691688449924"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:01.569494 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.569470 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6a9a9c-bd88-4ce1-a661-691688449924-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:01.570890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.570865 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-metrics-certs" pod "4c6a9a9c-bd88-4ce1-a661-691688449924" (UID: "4c6a9a9c-bd88-4ce1-a661-691688449924"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:01.571108 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.571083 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-maas-api-tls" pod "4c6a9a9c-bd88-4ce1-a661-691688449924" (UID: "4c6a9a9c-bd88-4ce1-a661-691688449924"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:01.571196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.571126 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6a9a9c-bd88-4ce1-a661-691688449924-kube-api-access-l959m" pod "4c6a9a9c-bd88-4ce1-a661-691688449924" (UID: "4c6a9a9c-bd88-4ce1-a661-691688449924"). InnerVolumeSpecName "kube-api-access-l959m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:01.670719 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.670688 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:01.670719 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.670713 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6a9a9c-bd88-4ce1-a661-691688449924-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:01.670719 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.670725 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l959m\" (UniqueName: \"kubernetes.io/projected/4c6a9a9c-bd88-4ce1-a661-691688449924-kube-api-access-l959m\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:01.697623 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:01.697555 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:02.096640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.096609 4711 generic.go:356] "Generic (PLEG): container finished" podID="4c6a9a9c-bd88-4ce1-a661-691688449924" containerID="c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969" exitCode=0 Nov 03 21:04:02.096820 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.096649 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" Nov 03 21:04:02.096820 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.096661 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" event={"ID":"4c6a9a9c-bd88-4ce1-a661-691688449924","Type":"ContainerDied","Data":"c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969"} Nov 03 21:04:02.096820 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.096708 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn" event={"ID":"4c6a9a9c-bd88-4ce1-a661-691688449924","Type":"ContainerDied","Data":"84d4fd7c6f34f88786b1f699ac1b0e07d0b8b477072750061781aeda05306b52"} Nov 03 21:04:02.096820 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.096730 4711 scope.go:122] "RemoveContainer" containerID="c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969" Nov 03 21:04:02.106825 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.106808 4711 scope.go:122] "RemoveContainer" containerID="c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969" Nov 03 21:04:02.107084 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:02.107063 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969\": container with ID starting with c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969 not found: ID does not exist" containerID="c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969" Nov 03 21:04:02.107161 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.107096 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969"} err="failed to get container status \"c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969\": rpc error: code = NotFound desc = could not find container \"c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969\": container with ID starting with c0c8261250f401b9a4b9fe61e526e58b72b010168a1658afae88b73a162ed969 not found: ID does not exist" Nov 03 21:04:02.120901 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.120877 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn"] Nov 03 21:04:02.127057 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.127036 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c7aecb-79b65577bb-g8wzn"] Nov 03 21:04:02.409798 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.409674 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:02.679651 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:02.679576 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6a9a9c-bd88-4ce1-a661-691688449924" path="/var/lib/kubelet/pods/4c6a9a9c-bd88-4ce1-a661-691688449924/volumes" Nov 03 21:04:03.218166 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:03.218129 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:03.350913 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:03.350879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:03.907037 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:03.907003 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:04.133281 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:04.133246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:04.341276 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:04.341242 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:04.490746 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:04.490708 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:04.723265 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:04.723156 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:05.336916 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:05.336883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:05.524815 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:05.524782 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:05.791315 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:05.791280 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:06.019697 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:06.019662 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp"] Nov 03 21:04:06.019950 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:06.019920 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" containerID="cri-o://c6413d1816474e9cc808d8db59ba69deda2f1a242a9cba0c22a9fe777bf8338e" gracePeriod=30 Nov 03 21:04:06.113646 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:06.113549 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:06.612022 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:06.611952 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:07.395029 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:07.394989 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:08.290793 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:08.290756 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:09.097104 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:09.097051 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:09.236208 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:09.236170 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:09.688545 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:09.688504 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:10.017861 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:10.017826 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:10.178009 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:10.177968 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:10.436804 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:10.436706 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:10.662303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:10.662269 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:10.947522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:10.947487 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:11.150601 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.150570 4711 generic.go:356] "Generic (PLEG): container finished" podID="7adfd185-9df3-44db-afda-b5c67a858d54" containerID="c6413d1816474e9cc808d8db59ba69deda2f1a242a9cba0c22a9fe777bf8338e" exitCode=0 Nov 03 21:04:11.150789 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.150626 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" event={"ID":"7adfd185-9df3-44db-afda-b5c67a858d54","Type":"ContainerDied","Data":"c6413d1816474e9cc808d8db59ba69deda2f1a242a9cba0c22a9fe777bf8338e"} Nov 03 21:04:11.290646 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.290622 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 21:04:11.380752 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.380713 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-data\") pod \"7adfd185-9df3-44db-afda-b5c67a858d54\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " Nov 03 21:04:11.380941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.380768 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7adfd185-9df3-44db-afda-b5c67a858d54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adfd185-9df3-44db-afda-b5c67a858d54-istiod-ca-cert\") pod \"7adfd185-9df3-44db-afda-b5c67a858d54\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " Nov 03 21:04:11.380941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.380803 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-istio-token\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-istio-token\") pod \"7adfd185-9df3-44db-afda-b5c67a858d54\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " Nov 03 21:04:11.380941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.380835 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-credential-socket\") pod \"7adfd185-9df3-44db-afda-b5c67a858d54\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " Nov 03 21:04:11.381133 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381008 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-envoy\") pod \"7adfd185-9df3-44db-afda-b5c67a858d54\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " Nov 03 21:04:11.381133 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381067 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7adfd185-9df3-44db-afda-b5c67a858d54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adfd185-9df3-44db-afda-b5c67a858d54-istio-podinfo\") pod \"7adfd185-9df3-44db-afda-b5c67a858d54\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " Nov 03 21:04:11.381133 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381100 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-socket\") pod \"7adfd185-9df3-44db-afda-b5c67a858d54\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " Nov 03 21:04:11.381133 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381089 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-credential-socket" pod "7adfd185-9df3-44db-afda-b5c67a858d54" (UID: "7adfd185-9df3-44db-afda-b5c67a858d54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:11.381331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381134 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-data" pod "7adfd185-9df3-44db-afda-b5c67a858d54" (UID: "7adfd185-9df3-44db-afda-b5c67a858d54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:11.381331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381243 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-certs\") pod \"7adfd185-9df3-44db-afda-b5c67a858d54\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " Nov 03 21:04:11.381331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381259 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adfd185-9df3-44db-afda-b5c67a858d54-istiod-ca-cert" pod "7adfd185-9df3-44db-afda-b5c67a858d54" (UID: "7adfd185-9df3-44db-afda-b5c67a858d54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:11.381331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381278 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-kube-api-access-ghpbj\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-kube-api-access-ghpbj\") pod \"7adfd185-9df3-44db-afda-b5c67a858d54\" (UID: \"7adfd185-9df3-44db-afda-b5c67a858d54\") " Nov 03 21:04:11.381602 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381383 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-socket" pod "7adfd185-9df3-44db-afda-b5c67a858d54" (UID: "7adfd185-9df3-44db-afda-b5c67a858d54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:11.381602 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381571 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-certs" pod "7adfd185-9df3-44db-afda-b5c67a858d54" (UID: "7adfd185-9df3-44db-afda-b5c67a858d54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:11.381786 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381768 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.381860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381787 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.381860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381799 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.381860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381811 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.381860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.381825 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adfd185-9df3-44db-afda-b5c67a858d54-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.383154 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.383129 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-istio-token" pod "7adfd185-9df3-44db-afda-b5c67a858d54" (UID: "7adfd185-9df3-44db-afda-b5c67a858d54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:11.383300 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.383279 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-kube-api-access-ghpbj" pod "7adfd185-9df3-44db-afda-b5c67a858d54" (UID: "7adfd185-9df3-44db-afda-b5c67a858d54"). InnerVolumeSpecName "kube-api-access-ghpbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:11.383583 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.383558 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7adfd185-9df3-44db-afda-b5c67a858d54-istio-podinfo" pod "7adfd185-9df3-44db-afda-b5c67a858d54" (UID: "7adfd185-9df3-44db-afda-b5c67a858d54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:04:11.383703 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.383623 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-envoy" pod "7adfd185-9df3-44db-afda-b5c67a858d54" (UID: "7adfd185-9df3-44db-afda-b5c67a858d54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:11.483407 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.483368 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghpbj\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-kube-api-access-ghpbj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.483407 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.483400 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adfd185-9df3-44db-afda-b5c67a858d54-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.483407 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.483412 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adfd185-9df3-44db-afda-b5c67a858d54-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.483691 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.483420 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adfd185-9df3-44db-afda-b5c67a858d54-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:11.824950 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:11.824909 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:12.158034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:12.157944 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" Nov 03 21:04:12.158034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:12.157958 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp" event={"ID":"7adfd185-9df3-44db-afda-b5c67a858d54","Type":"ContainerDied","Data":"8e5d29a261f9255710e840393be35d6b99b90b11eb3ebafa42b41689c85e7071"} Nov 03 21:04:12.158034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:12.158021 4711 scope.go:122] "RemoveContainer" containerID="c6413d1816474e9cc808d8db59ba69deda2f1a242a9cba0c22a9fe777bf8338e" Nov 03 21:04:12.181570 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:12.181520 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp"] Nov 03 21:04:12.184512 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:12.184488 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4f93d1-openshift-default-66d47948fc-k8hpp"] Nov 03 21:04:12.407150 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:12.407111 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:12.551285 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:12.551245 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:12.679215 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:12.679180 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" path="/var/lib/kubelet/pods/7adfd185-9df3-44db-afda-b5c67a858d54/volumes" Nov 03 21:04:12.726524 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:12.726423 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:13.669207 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:13.669159 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:13.852072 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:13.852033 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk"] Nov 03 21:04:13.921742 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:13.921628 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:14.357039 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:14.356988 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:14.488205 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:14.488164 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:15.213854 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:15.213714 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:15.424396 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:15.424362 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht"] Nov 03 21:04:15.434878 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:15.434851 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh"] Nov 03 21:04:15.556821 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:15.556786 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:16.008981 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:16.008949 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:16.707322 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:16.707275 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:17.412179 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:17.412140 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:17.531379 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:17.531341 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:04:18.206141 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:18.206100 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:18.912879 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:18.912825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:19.810612 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:19.810573 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:20.395469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:20.395412 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:20.598975 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:20.598930 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:20.736170 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:20.736077 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:21.280987 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:21.280934 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:21.484527 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:21.484496 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:22.046106 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:22.046069 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:22.330584 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:22.330493 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:22.643227 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:22.643136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:23.074820 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:23.074775 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:23.799722 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:23.799678 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:24.505153 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:24.505114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:25.205924 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:25.205879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:25.906088 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:25.906043 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:26.753735 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:26.753700 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:27.113351 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:27.113314 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:27.250589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:27.250547 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:27.505270 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:27.505240 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:27.705869 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:27.705826 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:27.852593 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:27.852500 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc"] Nov 03 21:04:27.853071 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:27.852755 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" containerID="cri-o://412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8" gracePeriod=30 Nov 03 21:04:27.953911 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:27.953873 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:28.319674 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:28.319627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:28.695672 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:28.695586 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:28.876004 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:28.875943 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" podUID="e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" containerName="main" containerID="cri-o://4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59" gracePeriod=45 Nov 03 21:04:29.119846 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.119807 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:29.143883 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.143855 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:04:29.188360 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.188325 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-kube-api-access-j8nqq\" (UniqueName: \"kubernetes.io/projected/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-kube-api-access-j8nqq\") pod \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " Nov 03 21:04:29.188582 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.188385 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tmp-dir\") pod \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " Nov 03 21:04:29.188582 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.188429 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-model-cache\") pod \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " Nov 03 21:04:29.188718 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.188634 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-home\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-home\") pod \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " Nov 03 21:04:29.188782 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.188722 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tls-certs\") pod \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " Nov 03 21:04:29.188782 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.188729 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-model-cache" pod "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" (UID: "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:29.188782 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.188754 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tmp-dir" pod "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" (UID: "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:29.188953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.188797 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-dshm\") pod \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\" (UID: \"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a\") " Nov 03 21:04:29.188953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.188919 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-home" pod "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" (UID: "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:29.189271 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.189245 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:29.189271 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.189272 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:29.189440 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.189286 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:29.191076 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.191055 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-kube-api-access-j8nqq" pod "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" (UID: "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a"). InnerVolumeSpecName "kube-api-access-j8nqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:29.191485 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.191456 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-dshm" pod "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" (UID: "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:29.191485 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.191469 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tls-certs" pod "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" (UID: "e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:29.230592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.230563 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt"] Nov 03 21:04:29.243102 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.243032 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc"] Nov 03 21:04:29.266521 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.266483 4711 generic.go:356] "Generic (PLEG): container finished" podID="e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" containerID="4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59" exitCode=0 Nov 03 21:04:29.266703 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.266593 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" event={"ID":"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a","Type":"ContainerDied","Data":"4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59"} Nov 03 21:04:29.266703 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.266639 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" event={"ID":"e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a","Type":"ContainerDied","Data":"514e3413cd0a1e98f640c8cf117aa9343eeab92eb469fc0f278566dce4842967"} Nov 03 21:04:29.266703 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.266659 4711 scope.go:122] "RemoveContainer" containerID="4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59" Nov 03 21:04:29.266826 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.266807 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk" Nov 03 21:04:29.283549 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.283525 4711 scope.go:122] "RemoveContainer" containerID="4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59" Nov 03 21:04:29.283880 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:29.283851 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59\": container with ID starting with 4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59 not found: ID does not exist" containerID="4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59" Nov 03 21:04:29.283957 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.283895 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59"} err="failed to get container status \"4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59\": rpc error: code = NotFound desc = could not find container \"4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59\": container with ID starting with 4359689a25732ea4faf10b8b3540b858587fc80359ad5ade396bc7a0a1d90d59 not found: ID does not exist" Nov 03 21:04:29.290514 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.290488 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:29.290514 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.290514 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8nqq\" (UniqueName: \"kubernetes.io/projected/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-kube-api-access-j8nqq\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:29.290727 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.290523 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:29.295124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.295091 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk"] Nov 03 21:04:29.297707 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.297681 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-99a8fe/sub-test-model-99a8fe-kserve-6cf8b5f7f7-p27fk"] Nov 03 21:04:29.343687 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.343654 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:29.473939 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.473887 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:29.717817 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.717778 4711 patch_prober.go:29] interesting pod/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:04:29.717973 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.717843 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:04:29.909519 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:29.909484 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:30.109766 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.109731 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:30.445024 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.444930 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" containerID="cri-o://f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7" gracePeriod=45 Nov 03 21:04:30.456984 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.456942 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" containerID="cri-o://6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0" gracePeriod=45 Nov 03 21:04:30.609360 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.609315 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:30.681902 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.681863 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" path="/var/lib/kubelet/pods/e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a/volumes" Nov 03 21:04:30.736827 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.736804 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 21:04:30.739231 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.739212 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 21:04:30.809524 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.809485 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01de89c8-912b-4b07-857f-c03e11f08d89-tls-certs\" (UniqueName: \"kubernetes.io/secret/01de89c8-912b-4b07-857f-c03e11f08d89-tls-certs\") pod \"01de89c8-912b-4b07-857f-c03e11f08d89\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " Nov 03 21:04:30.809710 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.809543 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f1fee8-6816-47ac-98e3-073ff518cdca-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f1fee8-6816-47ac-98e3-073ff518cdca-tls-certs\") pod \"f2f1fee8-6816-47ac-98e3-073ff518cdca\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " Nov 03 21:04:30.809710 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.809586 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-tmp-dir\") pod \"f2f1fee8-6816-47ac-98e3-073ff518cdca\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " Nov 03 21:04:30.809710 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.809616 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-model-cache\") pod \"f2f1fee8-6816-47ac-98e3-073ff518cdca\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " Nov 03 21:04:30.809710 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.809654 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-home\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-home\") pod \"f2f1fee8-6816-47ac-98e3-073ff518cdca\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " Nov 03 21:04:30.809956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.809729 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-dshm\") pod \"f2f1fee8-6816-47ac-98e3-073ff518cdca\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " Nov 03 21:04:30.809956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.809801 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01de89c8-912b-4b07-857f-c03e11f08d89-kube-api-access-k65fk\" (UniqueName: \"kubernetes.io/projected/01de89c8-912b-4b07-857f-c03e11f08d89-kube-api-access-k65fk\") pod \"01de89c8-912b-4b07-857f-c03e11f08d89\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " Nov 03 21:04:30.810261 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.809872 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-tmp-dir\") pod \"01de89c8-912b-4b07-857f-c03e11f08d89\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " Nov 03 21:04:30.810626 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.810305 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-model-cache\") pod \"01de89c8-912b-4b07-857f-c03e11f08d89\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " Nov 03 21:04:30.810739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.810650 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f1fee8-6816-47ac-98e3-073ff518cdca-kube-api-access-vwxdx\" (UniqueName: \"kubernetes.io/projected/f2f1fee8-6816-47ac-98e3-073ff518cdca-kube-api-access-vwxdx\") pod \"f2f1fee8-6816-47ac-98e3-073ff518cdca\" (UID: \"f2f1fee8-6816-47ac-98e3-073ff518cdca\") " Nov 03 21:04:30.810739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.809887 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-tmp-dir" pod "f2f1fee8-6816-47ac-98e3-073ff518cdca" (UID: "f2f1fee8-6816-47ac-98e3-073ff518cdca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.810739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.810118 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-model-cache" pod "f2f1fee8-6816-47ac-98e3-073ff518cdca" (UID: "f2f1fee8-6816-47ac-98e3-073ff518cdca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.810739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.810120 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-tmp-dir" pod "01de89c8-912b-4b07-857f-c03e11f08d89" (UID: "01de89c8-912b-4b07-857f-c03e11f08d89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.810739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.810166 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-home" pod "f2f1fee8-6816-47ac-98e3-073ff518cdca" (UID: "f2f1fee8-6816-47ac-98e3-073ff518cdca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.810739 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.810579 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-model-cache" pod "01de89c8-912b-4b07-857f-c03e11f08d89" (UID: "01de89c8-912b-4b07-857f-c03e11f08d89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.811110 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.811084 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-home\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-home\") pod \"01de89c8-912b-4b07-857f-c03e11f08d89\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " Nov 03 21:04:30.811362 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.811340 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-home" pod "01de89c8-912b-4b07-857f-c03e11f08d89" (UID: "01de89c8-912b-4b07-857f-c03e11f08d89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.811509 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.811489 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-dshm\") pod \"01de89c8-912b-4b07-857f-c03e11f08d89\" (UID: \"01de89c8-912b-4b07-857f-c03e11f08d89\") " Nov 03 21:04:30.812290 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.812265 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.812395 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.812293 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.812395 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.812309 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.812395 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.812323 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.812395 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.812336 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.812395 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.812348 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.813532 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.813491 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01de89c8-912b-4b07-857f-c03e11f08d89-tls-certs" pod "01de89c8-912b-4b07-857f-c03e11f08d89" (UID: "01de89c8-912b-4b07-857f-c03e11f08d89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:30.813634 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.813564 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-dshm" pod "01de89c8-912b-4b07-857f-c03e11f08d89" (UID: "01de89c8-912b-4b07-857f-c03e11f08d89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.813634 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.813618 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f1fee8-6816-47ac-98e3-073ff518cdca-tls-certs" pod "f2f1fee8-6816-47ac-98e3-073ff518cdca" (UID: "f2f1fee8-6816-47ac-98e3-073ff518cdca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:30.813720 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.813680 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01de89c8-912b-4b07-857f-c03e11f08d89-kube-api-access-k65fk" pod "01de89c8-912b-4b07-857f-c03e11f08d89" (UID: "01de89c8-912b-4b07-857f-c03e11f08d89"). InnerVolumeSpecName "kube-api-access-k65fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:30.813863 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.813831 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-dshm" pod "f2f1fee8-6816-47ac-98e3-073ff518cdca" (UID: "f2f1fee8-6816-47ac-98e3-073ff518cdca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:30.814657 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.814635 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f1fee8-6816-47ac-98e3-073ff518cdca-kube-api-access-vwxdx" pod "f2f1fee8-6816-47ac-98e3-073ff518cdca" (UID: "f2f1fee8-6816-47ac-98e3-073ff518cdca"). InnerVolumeSpecName "kube-api-access-vwxdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:30.913194 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.913161 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01de89c8-912b-4b07-857f-c03e11f08d89-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.913194 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.913189 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01de89c8-912b-4b07-857f-c03e11f08d89-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.913194 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.913199 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f1fee8-6816-47ac-98e3-073ff518cdca-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.913661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.913208 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f1fee8-6816-47ac-98e3-073ff518cdca-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.913661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.913217 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65fk\" (UniqueName: \"kubernetes.io/projected/01de89c8-912b-4b07-857f-c03e11f08d89-kube-api-access-k65fk\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:30.913661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:30.913227 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwxdx\" (UniqueName: \"kubernetes.io/projected/f2f1fee8-6816-47ac-98e3-073ff518cdca-kube-api-access-vwxdx\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:31.279755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.279721 4711 generic.go:356] "Generic (PLEG): container finished" podID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerID="f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7" exitCode=0 Nov 03 21:04:31.279968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.279764 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" event={"ID":"f2f1fee8-6816-47ac-98e3-073ff518cdca","Type":"ContainerDied","Data":"f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7"} Nov 03 21:04:31.279968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.279816 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" event={"ID":"f2f1fee8-6816-47ac-98e3-073ff518cdca","Type":"ContainerDied","Data":"f292afe06c4d965bfeb3f6cd5f8f79d68793aedd31572aa881789808ef7cbbd0"} Nov 03 21:04:31.279968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.279774 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht" Nov 03 21:04:31.279968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.279838 4711 scope.go:122] "RemoveContainer" containerID="f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7" Nov 03 21:04:31.281807 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.281778 4711 generic.go:356] "Generic (PLEG): container finished" podID="01de89c8-912b-4b07-857f-c03e11f08d89" containerID="6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0" exitCode=0 Nov 03 21:04:31.281937 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.281816 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" event={"ID":"01de89c8-912b-4b07-857f-c03e11f08d89","Type":"ContainerDied","Data":"6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0"} Nov 03 21:04:31.281937 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.281873 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" event={"ID":"01de89c8-912b-4b07-857f-c03e11f08d89","Type":"ContainerDied","Data":"04a83213cbfadf1791418d37204977a0549bf4da2b29706188a0302447ded338"} Nov 03 21:04:31.281937 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.281826 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh" Nov 03 21:04:31.291395 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.291370 4711 scope.go:122] "RemoveContainer" containerID="f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7" Nov 03 21:04:31.291740 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:31.291723 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7\": container with ID starting with f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7 not found: ID does not exist" containerID="f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7" Nov 03 21:04:31.291808 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.291748 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7"} err="failed to get container status \"f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7\": rpc error: code = NotFound desc = could not find container \"f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7\": container with ID starting with f2c4190646c964758f2a12353ea59d26c108f7e1eddab8a5564544dc9e25d8d7 not found: ID does not exist" Nov 03 21:04:31.291808 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.291767 4711 scope.go:122] "RemoveContainer" containerID="6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0" Nov 03 21:04:31.302885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.302859 4711 scope.go:122] "RemoveContainer" containerID="6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0" Nov 03 21:04:31.304281 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:31.304250 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0\": container with ID starting with 6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0 not found: ID does not exist" containerID="6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0" Nov 03 21:04:31.304387 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.304289 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0"} err="failed to get container status \"6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0\": rpc error: code = NotFound desc = could not find container \"6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0\": container with ID starting with 6d43a042061f7e905e47462a7fa6579c58e20c601b1377b20b90c50fa15269b0 not found: ID does not exist" Nov 03 21:04:31.305909 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.305891 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht"] Nov 03 21:04:31.308379 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.308350 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:31.312120 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.312085 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-085566/distinct-w2-085566-kserve-69df6b575c-x87ht"] Nov 03 21:04:31.324300 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.324267 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh"] Nov 03 21:04:31.328513 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:31.328482 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-085566/premium-w2-085566-kserve-6c687f56d6-5n2bh"] Nov 03 21:04:32.100070 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:32.100020 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:32.679934 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:32.679901 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" path="/var/lib/kubelet/pods/01de89c8-912b-4b07-857f-c03e11f08d89/volumes" Nov 03 21:04:32.680350 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:32.680336 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" path="/var/lib/kubelet/pods/f2f1fee8-6816-47ac-98e3-073ff518cdca/volumes" Nov 03 21:04:32.799836 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:32.799795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:33.126915 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.126888 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 21:04:33.237794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.237703 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-socket\") pod \"4153e58d-b2c1-48c5-9175-984b7115edb5\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " Nov 03 21:04:33.237794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.237748 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4153e58d-b2c1-48c5-9175-984b7115edb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4153e58d-b2c1-48c5-9175-984b7115edb5-istiod-ca-cert\") pod \"4153e58d-b2c1-48c5-9175-984b7115edb5\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " Nov 03 21:04:33.237794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.237778 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-podinfo\") pod \"4153e58d-b2c1-48c5-9175-984b7115edb5\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " Nov 03 21:04:33.238076 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.237899 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-credential-socket\") pod \"4153e58d-b2c1-48c5-9175-984b7115edb5\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " Nov 03 21:04:33.238076 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.237965 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-certs\") pod \"4153e58d-b2c1-48c5-9175-984b7115edb5\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " Nov 03 21:04:33.238076 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238002 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-token\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-token\") pod \"4153e58d-b2c1-48c5-9175-984b7115edb5\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " Nov 03 21:04:33.238076 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238011 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-socket" pod "4153e58d-b2c1-48c5-9175-984b7115edb5" (UID: "4153e58d-b2c1-48c5-9175-984b7115edb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:33.238076 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238049 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-data\") pod \"4153e58d-b2c1-48c5-9175-984b7115edb5\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " Nov 03 21:04:33.238344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238132 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-envoy\") pod \"4153e58d-b2c1-48c5-9175-984b7115edb5\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " Nov 03 21:04:33.238344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238166 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-kube-api-access-7fhwj\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-kube-api-access-7fhwj\") pod \"4153e58d-b2c1-48c5-9175-984b7115edb5\" (UID: \"4153e58d-b2c1-48c5-9175-984b7115edb5\") " Nov 03 21:04:33.238344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238134 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-credential-socket" pod "4153e58d-b2c1-48c5-9175-984b7115edb5" (UID: "4153e58d-b2c1-48c5-9175-984b7115edb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:33.238344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238181 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4153e58d-b2c1-48c5-9175-984b7115edb5-istiod-ca-cert" pod "4153e58d-b2c1-48c5-9175-984b7115edb5" (UID: "4153e58d-b2c1-48c5-9175-984b7115edb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:33.238344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238252 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-data" pod "4153e58d-b2c1-48c5-9175-984b7115edb5" (UID: "4153e58d-b2c1-48c5-9175-984b7115edb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:33.238632 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238537 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-certs" pod "4153e58d-b2c1-48c5-9175-984b7115edb5" (UID: "4153e58d-b2c1-48c5-9175-984b7115edb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:33.238757 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238738 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.238813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238763 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.238813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238779 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.238813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238793 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.238813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.238806 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4153e58d-b2c1-48c5-9175-984b7115edb5-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.240158 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.240128 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-token" pod "4153e58d-b2c1-48c5-9175-984b7115edb5" (UID: "4153e58d-b2c1-48c5-9175-984b7115edb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:33.240290 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.240263 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-envoy" pod "4153e58d-b2c1-48c5-9175-984b7115edb5" (UID: "4153e58d-b2c1-48c5-9175-984b7115edb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:33.240435 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.240411 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-podinfo" pod "4153e58d-b2c1-48c5-9175-984b7115edb5" (UID: "4153e58d-b2c1-48c5-9175-984b7115edb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:04:33.240800 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.240778 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-kube-api-access-7fhwj" pod "4153e58d-b2c1-48c5-9175-984b7115edb5" (UID: "4153e58d-b2c1-48c5-9175-984b7115edb5"). InnerVolumeSpecName "kube-api-access-7fhwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:33.300026 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.299991 4711 generic.go:356] "Generic (PLEG): container finished" podID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerID="412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8" exitCode=0 Nov 03 21:04:33.300217 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.300052 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" Nov 03 21:04:33.300217 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.300079 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" event={"ID":"4153e58d-b2c1-48c5-9175-984b7115edb5","Type":"ContainerDied","Data":"412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8"} Nov 03 21:04:33.300217 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.300147 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc" event={"ID":"4153e58d-b2c1-48c5-9175-984b7115edb5","Type":"ContainerDied","Data":"09432f0f0a6134878792b38026fca68fe004e57315f8141d086d30404d9b06a6"} Nov 03 21:04:33.300217 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.300177 4711 scope.go:122] "RemoveContainer" containerID="412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8" Nov 03 21:04:33.310354 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.310338 4711 scope.go:122] "RemoveContainer" containerID="e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0" Nov 03 21:04:33.319099 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.319076 4711 scope.go:122] "RemoveContainer" containerID="412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8" Nov 03 21:04:33.319407 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:33.319386 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8\": container with ID starting with 412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8 not found: ID does not exist" containerID="412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8" Nov 03 21:04:33.319527 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.319418 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8"} err="failed to get container status \"412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8\": rpc error: code = NotFound desc = could not find container \"412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8\": container with ID starting with 412651ade3d3be26ab62592778e3ff848d7fcccc0179bc2a759206b1cf4b61f8 not found: ID does not exist" Nov 03 21:04:33.319527 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.319460 4711 scope.go:122] "RemoveContainer" containerID="e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0" Nov 03 21:04:33.319764 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:33.319740 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0\": container with ID starting with e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0 not found: ID does not exist" containerID="e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0" Nov 03 21:04:33.319804 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.319774 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0"} err="failed to get container status \"e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0\": rpc error: code = NotFound desc = could not find container \"e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0\": container with ID starting with e99d3c45180c89dcedc82c9d4966a79d75d25eab92e910ab6abe30a4d7b6c3c0 not found: ID does not exist" Nov 03 21:04:33.323947 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.323924 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc"] Nov 03 21:04:33.327813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.327793 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-db5198-openshift-default-6bfbb87b9c-mlmrc"] Nov 03 21:04:33.340528 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.340503 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.340630 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.340529 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.340630 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.340545 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4153e58d-b2c1-48c5-9175-984b7115edb5-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.340630 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.340559 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fhwj\" (UniqueName: \"kubernetes.io/projected/4153e58d-b2c1-48c5-9175-984b7115edb5-kube-api-access-7fhwj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:33.491556 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:33.491439 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:34.230898 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.230854 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" podUID="2acc379e-4a07-4f2b-ac95-1f915baa79a9" containerName="payload-processing" containerID="cri-o://390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb" gracePeriod=25 Nov 03 21:04:34.243377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.243308 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" podUID="99d18155-451a-494b-ad61-9d56bf91ad32" containerName="payload-pre-processing" containerID="cri-o://9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0" gracePeriod=25 Nov 03 21:04:34.408436 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.408330 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:34.504749 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.504704 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 21:04:34.543217 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.543189 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 21:04:34.553749 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.553722 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acc379e-4a07-4f2b-ac95-1f915baa79a9-kube-api-access-zg7fc\" (UniqueName: \"kubernetes.io/projected/2acc379e-4a07-4f2b-ac95-1f915baa79a9-kube-api-access-zg7fc\") pod \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\" (UID: \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\") " Nov 03 21:04:34.553933 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.553821 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2acc379e-4a07-4f2b-ac95-1f915baa79a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2acc379e-4a07-4f2b-ac95-1f915baa79a9-plugins-config-volume\") pod \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\" (UID: \"2acc379e-4a07-4f2b-ac95-1f915baa79a9\") " Nov 03 21:04:34.554230 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.554202 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acc379e-4a07-4f2b-ac95-1f915baa79a9-plugins-config-volume" pod "2acc379e-4a07-4f2b-ac95-1f915baa79a9" (UID: "2acc379e-4a07-4f2b-ac95-1f915baa79a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:34.555861 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.555836 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acc379e-4a07-4f2b-ac95-1f915baa79a9-kube-api-access-zg7fc" pod "2acc379e-4a07-4f2b-ac95-1f915baa79a9" (UID: "2acc379e-4a07-4f2b-ac95-1f915baa79a9"). InnerVolumeSpecName "kube-api-access-zg7fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:34.655538 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.655499 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d18155-451a-494b-ad61-9d56bf91ad32-kube-api-access-9qbgj\" (UniqueName: \"kubernetes.io/projected/99d18155-451a-494b-ad61-9d56bf91ad32-kube-api-access-9qbgj\") pod \"99d18155-451a-494b-ad61-9d56bf91ad32\" (UID: \"99d18155-451a-494b-ad61-9d56bf91ad32\") " Nov 03 21:04:34.655705 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.655565 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d18155-451a-494b-ad61-9d56bf91ad32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d18155-451a-494b-ad61-9d56bf91ad32-plugins-config-volume\") pod \"99d18155-451a-494b-ad61-9d56bf91ad32\" (UID: \"99d18155-451a-494b-ad61-9d56bf91ad32\") " Nov 03 21:04:34.655939 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.655912 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d18155-451a-494b-ad61-9d56bf91ad32-plugins-config-volume" pod "99d18155-451a-494b-ad61-9d56bf91ad32" (UID: "99d18155-451a-494b-ad61-9d56bf91ad32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:34.655939 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.655916 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg7fc\" (UniqueName: \"kubernetes.io/projected/2acc379e-4a07-4f2b-ac95-1f915baa79a9-kube-api-access-zg7fc\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.656103 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.655963 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2acc379e-4a07-4f2b-ac95-1f915baa79a9-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.657634 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.657614 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d18155-451a-494b-ad61-9d56bf91ad32-kube-api-access-9qbgj" pod "99d18155-451a-494b-ad61-9d56bf91ad32" (UID: "99d18155-451a-494b-ad61-9d56bf91ad32"). InnerVolumeSpecName "kube-api-access-9qbgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:34.680086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.680055 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" path="/var/lib/kubelet/pods/4153e58d-b2c1-48c5-9175-984b7115edb5/volumes" Nov 03 21:04:34.757526 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.757503 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qbgj\" (UniqueName: \"kubernetes.io/projected/99d18155-451a-494b-ad61-9d56bf91ad32-kube-api-access-9qbgj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.757526 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:34.757526 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d18155-451a-494b-ad61-9d56bf91ad32-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:35.314258 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.314226 4711 generic.go:356] "Generic (PLEG): container finished" podID="99d18155-451a-494b-ad61-9d56bf91ad32" containerID="9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0" exitCode=0 Nov 03 21:04:35.314777 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.314283 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" Nov 03 21:04:35.314777 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.314318 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" event={"ID":"99d18155-451a-494b-ad61-9d56bf91ad32","Type":"ContainerDied","Data":"9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0"} Nov 03 21:04:35.314777 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.314383 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc" event={"ID":"99d18155-451a-494b-ad61-9d56bf91ad32","Type":"ContainerDied","Data":"07df225904e7c693268c946bc4346bf0aa310acbe2d0f1b396f34a9d7ba855ee"} Nov 03 21:04:35.314777 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.314413 4711 scope.go:122] "RemoveContainer" containerID="9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0" Nov 03 21:04:35.316043 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.316021 4711 generic.go:356] "Generic (PLEG): container finished" podID="2acc379e-4a07-4f2b-ac95-1f915baa79a9" containerID="390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb" exitCode=0 Nov 03 21:04:35.316141 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.316055 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" event={"ID":"2acc379e-4a07-4f2b-ac95-1f915baa79a9","Type":"ContainerDied","Data":"390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb"} Nov 03 21:04:35.316141 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.316070 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" Nov 03 21:04:35.316141 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.316095 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt" event={"ID":"2acc379e-4a07-4f2b-ac95-1f915baa79a9","Type":"ContainerDied","Data":"7397314f041eb2e4e0e3c3161fa0429ed8ec13f6364735e458d22afa32784339"} Nov 03 21:04:35.323844 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.323826 4711 scope.go:122] "RemoveContainer" containerID="9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0" Nov 03 21:04:35.324144 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:35.324115 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0\": container with ID starting with 9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0 not found: ID does not exist" containerID="9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0" Nov 03 21:04:35.324244 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.324150 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0"} err="failed to get container status \"9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0\": rpc error: code = NotFound desc = could not find container \"9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0\": container with ID starting with 9cbec7157b90be2c19f1c0befe3d91cb1c50475e54f65a449d7b7e74d30558d0 not found: ID does not exist" Nov 03 21:04:35.324244 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.324170 4711 scope.go:122] "RemoveContainer" containerID="390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb" Nov 03 21:04:35.333419 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.333401 4711 scope.go:122] "RemoveContainer" containerID="390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb" Nov 03 21:04:35.333751 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:35.333732 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb\": container with ID starting with 390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb not found: ID does not exist" containerID="390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb" Nov 03 21:04:35.333799 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.333762 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb"} err="failed to get container status \"390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb\": rpc error: code = NotFound desc = could not find container \"390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb\": container with ID starting with 390127ef66cd70f0c7e1d40216b13b4ef598df0e0ac2d0792cfb6c16e5c9fdcb not found: ID does not exist" Nov 03 21:04:35.396071 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.396039 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt"] Nov 03 21:04:35.423234 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.423198 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-085566-64777db978-9bjqt"] Nov 03 21:04:35.435195 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.435163 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc"] Nov 03 21:04:35.444790 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.444756 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-085566-bd6dc9bc5-dwwdc"] Nov 03 21:04:35.467830 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.467795 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:35.983347 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:35.983313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:36.116856 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:36.116810 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:36.271497 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:36.271463 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:36.484473 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:36.484403 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:36.609498 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:36.609394 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:36.679145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:36.679114 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acc379e-4a07-4f2b-ac95-1f915baa79a9" path="/var/lib/kubelet/pods/2acc379e-4a07-4f2b-ac95-1f915baa79a9/volumes" Nov 03 21:04:36.679554 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:36.679538 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d18155-451a-494b-ad61-9d56bf91ad32" path="/var/lib/kubelet/pods/99d18155-451a-494b-ad61-9d56bf91ad32/volumes" Nov 03 21:04:36.793637 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:36.793601 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:37.303779 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:37.303718 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:37.612397 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:37.612291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:38.299793 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:38.299738 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:38.997010 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:38.996976 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:39.705156 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:39.705114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:40.505325 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:40.505291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:41.314085 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:41.314046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:42.009804 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:42.009774 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:42.702419 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:42.702384 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:43.365611 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:43.365573 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:43.504803 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:43.504729 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:43.851918 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:43.851884 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:44.131510 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:44.131403 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk"] Nov 03 21:04:44.131708 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:44.131676 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" containerID="cri-o://791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8" gracePeriod=30 Nov 03 21:04:44.141496 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:44.141466 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:44.283115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:44.283077 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:44.416605 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:44.416512 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:44.941517 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:44.941475 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:45.079592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:45.079556 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:45.809441 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:45.809387 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:46.509218 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:46.509137 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:47.198930 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:47.198888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:47.530748 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:47.530716 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:04:47.543597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:47.543559 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:48.402433 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.402392 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:48.775952 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.775916 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69"] Nov 03 21:04:48.776481 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776466 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ecd9ea94-bb2d-44f1-8feb-c68a800f208d" containerName="maas-api" Nov 03 21:04:48.776547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776482 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd9ea94-bb2d-44f1-8feb-c68a800f208d" containerName="maas-api" Nov 03 21:04:48.776547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776495 4711 cpu_manager.go:414] "containerMap: removing container" podUID="60a690b0-4534-4550-93dd-313405071b7e" containerName="maas-api" Nov 03 21:04:48.776547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776500 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a690b0-4534-4550-93dd-313405071b7e" containerName="maas-api" Nov 03 21:04:48.776547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776511 4711 cpu_manager.go:414] "containerMap: removing container" podUID="0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" containerName="maas-api" Nov 03 21:04:48.776547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776517 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" containerName="maas-api" Nov 03 21:04:48.776547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776526 4711 cpu_manager.go:414] "containerMap: removing container" podUID="c13f2e6b-98a3-4200-be21-b6b28db4d38e" containerName="payload-processing" Nov 03 21:04:48.776547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776533 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13f2e6b-98a3-4200-be21-b6b28db4d38e" containerName="payload-processing" Nov 03 21:04:48.776547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776546 4711 cpu_manager.go:414] "containerMap: removing container" podUID="0289e060-4030-4262-8f32-2d4d656e71c0" containerName="maas-api" Nov 03 21:04:48.776547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776551 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0289e060-4030-4262-8f32-2d4d656e71c0" containerName="maas-api" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776562 4711 cpu_manager.go:414] "containerMap: removing container" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776567 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776573 4711 cpu_manager.go:414] "containerMap: removing container" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776578 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776585 4711 cpu_manager.go:414] "containerMap: removing container" podUID="99d18155-451a-494b-ad61-9d56bf91ad32" containerName="payload-pre-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776590 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d18155-451a-494b-ad61-9d56bf91ad32" containerName="payload-pre-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776598 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776602 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776612 4711 cpu_manager.go:414] "containerMap: removing container" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776619 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776626 4711 cpu_manager.go:414] "containerMap: removing container" podUID="40815cdd-d7d7-4471-a12c-e16fdccd9617" containerName="payload-pre-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776630 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40815cdd-d7d7-4471-a12c-e16fdccd9617" containerName="payload-pre-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776637 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9be03ba9-8f44-4bac-a84d-e91310aca456" containerName="payload-pre-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776641 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9be03ba9-8f44-4bac-a84d-e91310aca456" containerName="payload-pre-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776646 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4c6a9a9c-bd88-4ce1-a661-691688449924" containerName="maas-api" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776650 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6a9a9c-bd88-4ce1-a661-691688449924" containerName="maas-api" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776656 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b0599233-76dd-4eb9-9b5d-94a815dedd36" containerName="payload-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776661 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0599233-76dd-4eb9-9b5d-94a815dedd36" containerName="payload-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776668 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776672 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776682 4711 cpu_manager.go:414] "containerMap: removing container" podUID="c64d74d3-b7bd-491e-84db-07c449ae1946" containerName="payload-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776687 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64d74d3-b7bd-491e-84db-07c449ae1946" containerName="payload-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776698 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fcccaf40-5996-4050-bc42-c543a85cb8a6" containerName="maas-api" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776704 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcccaf40-5996-4050-bc42-c543a85cb8a6" containerName="maas-api" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776713 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerName="cleanup" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776718 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b75931b-bb2c-4415-9785-8bcf16418c61" containerName="cleanup" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776728 4711 cpu_manager.go:414] "containerMap: removing container" podUID="2acc379e-4a07-4f2b-ac95-1f915baa79a9" containerName="payload-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776732 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acc379e-4a07-4f2b-ac95-1f915baa79a9" containerName="payload-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776738 4711 cpu_manager.go:414] "containerMap: removing container" podUID="e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776742 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776756 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776760 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776772 4711 cpu_manager.go:414] "containerMap: removing container" podUID="739a3255-da88-41b3-bd81-331cdce0a6be" containerName="maas-api" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776780 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739a3255-da88-41b3-bd81-331cdce0a6be" containerName="maas-api" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776786 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776790 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776797 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776801 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776806 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776809 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776815 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776819 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776825 4711 cpu_manager.go:414] "containerMap: removing container" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776829 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776834 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerName="cleanup" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776838 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerName="cleanup" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776844 4711 cpu_manager.go:414] "containerMap: removing container" podUID="68629248-0876-4c2b-afde-739373f15134" containerName="cleanup" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776852 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68629248-0876-4c2b-afde-739373f15134" containerName="cleanup" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776864 4711 cpu_manager.go:414] "containerMap: removing container" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776869 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776876 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7d2b99c0-6966-4d2e-aa40-19afba1be993" containerName="payload-pre-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776880 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2b99c0-6966-4d2e-aa40-19afba1be993" containerName="payload-pre-processing" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776886 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776891 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776896 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6826f58b-3c04-4873-a9e7-970211daf43b" containerName="discovery" Nov 03 21:04:48.776875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776901 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6826f58b-3c04-4873-a9e7-970211daf43b" containerName="discovery" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776910 4711 cpu_manager.go:414] "containerMap: removing container" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776914 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776923 4711 cpu_manager.go:414] "containerMap: removing container" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerName="istio-proxy" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776930 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerName="istio-proxy" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776943 4711 cpu_manager.go:414] "containerMap: removing container" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" containerName="cleanup" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.776948 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979e0405-c600-496d-8ca1-13f749ba72eb" containerName="cleanup" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777019 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="739a3255-da88-41b3-bd81-331cdce0a6be" containerName="maas-api" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777027 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda8d702-b23b-4df0-8540-febd71b7f7fa" containerName="main" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777033 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adfd185-9df3-44db-afda-b5c67a858d54" containerName="istio-proxy" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777040 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f1fee8-6816-47ac-98e3-073ff518cdca" containerName="main" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777047 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777052 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777056 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="68629248-0876-4c2b-afde-739373f15134" containerName="cleanup" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777061 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a4ea2c-bb5c-4324-994e-564a7a2761a8" containerName="main" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777066 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2b99c0-6966-4d2e-aa40-19afba1be993" containerName="payload-pre-processing" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777072 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9d0cb1-0a0f-497d-a6a2-483f8338466b" containerName="main" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777076 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6826f58b-3c04-4873-a9e7-970211daf43b" containerName="discovery" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777082 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ad5f3f-8ae3-4c11-804f-2eba281ec11d" containerName="main" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777090 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="996362b4-6f32-4168-b8de-2433bf0f5cf4" containerName="istio-proxy" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777100 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd9ea94-bb2d-44f1-8feb-c68a800f208d" containerName="maas-api" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777106 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a690b0-4534-4550-93dd-313405071b7e" containerName="maas-api" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777110 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf9dbbc-4dda-4af1-9dd6-2da60134f11d" containerName="maas-api" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777116 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13f2e6b-98a3-4200-be21-b6b28db4d38e" containerName="payload-processing" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777120 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="0289e060-4030-4262-8f32-2d4d656e71c0" containerName="maas-api" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777125 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="01de89c8-912b-4b07-857f-c03e11f08d89" containerName="main" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777129 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ed1ab0-b90d-4e18-a33c-40e7da0ce032" containerName="istio-proxy" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777134 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d18155-451a-494b-ad61-9d56bf91ad32" containerName="payload-pre-processing" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777139 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777144 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="159c4fe1-8080-4f38-b1d9-d67250329c9a" containerName="istio-proxy" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777149 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="40815cdd-d7d7-4471-a12c-e16fdccd9617" containerName="payload-pre-processing" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777154 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9be03ba9-8f44-4bac-a84d-e91310aca456" containerName="payload-pre-processing" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777159 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6a9a9c-bd88-4ce1-a661-691688449924" containerName="maas-api" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777163 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0599233-76dd-4eb9-9b5d-94a815dedd36" containerName="payload-processing" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777173 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64d74d3-b7bd-491e-84db-07c449ae1946" containerName="payload-processing" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777181 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcccaf40-5996-4050-bc42-c543a85cb8a6" containerName="maas-api" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777186 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bbd2d74-56ec-4af8-8abb-9e8a10e04eaa" containerName="cleanup" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777192 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acc379e-4a07-4f2b-ac95-1f915baa79a9" containerName="payload-processing" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777196 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4dfd3ff-7286-4fa6-83a1-4d249aa18d5a" containerName="main" Nov 03 21:04:48.778600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.777202 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58b4c01-2f45-4b9b-a76c-c6d31c100337" containerName="istio-proxy" Nov 03 21:04:48.780259 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.780242 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.788476 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.788434 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-085566-openshift-default-dockercfg-dt25g\"" Nov 03 21:04:48.790365 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.790316 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q"] Nov 03 21:04:48.791150 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.791129 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4153e58d-b2c1-48c5-9175-984b7115edb5" containerName="istio-proxy" Nov 03 21:04:48.794990 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.794971 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69"] Nov 03 21:04:48.795100 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.795065 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.810341 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.810310 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q"] Nov 03 21:04:48.814799 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.814771 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.814958 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.814805 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021ae1d1-add4-4c42-a51d-685573728d5a-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.814958 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.814850 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.815077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.814988 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.815077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815023 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.815193 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815086 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.815193 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815129 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27t6j\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-kube-api-access-27t6j\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.815193 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815146 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.815193 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815168 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.815408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815309 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-kube-api-access-chv27\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.815408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815336 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.815408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815353 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.815408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815373 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.815631 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815498 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021ae1d1-add4-4c42-a51d-685573728d5a-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.815631 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815590 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.815631 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815621 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.815797 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815689 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.815797 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.815747 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.916868 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.916836 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.916875 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021ae1d1-add4-4c42-a51d-685573728d5a-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.916902 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.916929 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.916977 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917003 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917041 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917070 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021ae1d1-add4-4c42-a51d-685573728d5a-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917117 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917196 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.917314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917214 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.917955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917526 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.917955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917604 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27t6j\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-kube-api-access-27t6j\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.917955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917643 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.917955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917673 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.917955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917686 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.917955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917710 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-kube-api-access-chv27\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.917955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917735 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.917955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.917764 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.918504 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.918120 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.918504 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.918224 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-certs\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.918504 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.918339 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.918682 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.918531 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.918682 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.918628 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.919279 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.919202 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021ae1d1-add4-4c42-a51d-685573728d5a-istiod-ca-cert\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.919680 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.919640 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.920309 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.920285 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-credential-socket\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.920778 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.920658 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.920897 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.920810 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021ae1d1-add4-4c42-a51d-685573728d5a-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.920947 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.920912 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-podinfo\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.921006 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.920914 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-data\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.921813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.921790 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-envoy\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.930855 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.930835 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.931062 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.931032 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-kube-api-access-chv27\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-hl29q\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:48.931301 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.931280 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-istio-token\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:48.931431 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:48.931411 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27t6j\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-kube-api-access-27t6j\") pod \"e2e-worker-w2-085566-openshift-default-9958546fc-mvc69\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:49.094788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.094699 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:49.111987 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.111129 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:49.147302 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.147203 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:49.375787 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.375763 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 21:04:49.386024 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.385984 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69"] Nov 03 21:04:49.387763 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:04:49.387737 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021ae1d1_add4_4c42_a51d_685573728d5a.slice/crio-ed5a5b356b20e72452ec661d30dba41ee6c1f0f2c8fec673891ee15c1b8f2f3b WatchSource:0}: Error finding container ed5a5b356b20e72452ec661d30dba41ee6c1f0f2c8fec673891ee15c1b8f2f3b: Status 404 returned error can't find the container with id ed5a5b356b20e72452ec661d30dba41ee6c1f0f2c8fec673891ee15c1b8f2f3b Nov 03 21:04:49.389771 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.389736 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:04:49.389853 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.389801 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:04:49.389853 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.389830 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:04:49.395467 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.395417 4711 generic.go:356] "Generic (PLEG): container finished" podID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerID="791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8" exitCode=0 Nov 03 21:04:49.395562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.395488 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" Nov 03 21:04:49.395562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.395488 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" event={"ID":"581b6db0-0753-4059-8a03-e60bf2c3913b","Type":"ContainerDied","Data":"791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8"} Nov 03 21:04:49.395562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.395547 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk" event={"ID":"581b6db0-0753-4059-8a03-e60bf2c3913b","Type":"ContainerDied","Data":"47b7e1acc6f59ab9f03793115cf1a81870452c95863e95a99ac3a6ff17baf855"} Nov 03 21:04:49.395678 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.395578 4711 scope.go:122] "RemoveContainer" containerID="791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8" Nov 03 21:04:49.405649 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.405620 4711 scope.go:122] "RemoveContainer" containerID="791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8" Nov 03 21:04:49.406200 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:04:49.406166 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8\": container with ID starting with 791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8 not found: ID does not exist" containerID="791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8" Nov 03 21:04:49.406315 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.406214 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8"} err="failed to get container status \"791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8\": rpc error: code = NotFound desc = could not find container \"791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8\": container with ID starting with 791344fc3024ac62c9ea986f9238f87210f274cccc7c7993caa2a5c008577dc8 not found: ID does not exist" Nov 03 21:04:49.413721 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.413693 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q"] Nov 03 21:04:49.416316 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:04:49.416277 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d569bb_dbfc_4f88_95b3_cbf513d3ab75.slice/crio-b4a946697ddcce31ad69193b144da511eca4f6824dee44c3287044b83f275820 WatchSource:0}: Error finding container b4a946697ddcce31ad69193b144da511eca4f6824dee44c3287044b83f275820: Status 404 returned error can't find the container with id b4a946697ddcce31ad69193b144da511eca4f6824dee44c3287044b83f275820 Nov 03 21:04:49.421147 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.421116 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:04:49.421233 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.421177 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:04:49.421233 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.421207 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:04:49.423044 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423023 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-kube-api-access-bz4vb\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-kube-api-access-bz4vb\") pod \"581b6db0-0753-4059-8a03-e60bf2c3913b\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " Nov 03 21:04:49.423163 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423061 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-certs\") pod \"581b6db0-0753-4059-8a03-e60bf2c3913b\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " Nov 03 21:04:49.423163 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423084 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-podinfo\") pod \"581b6db0-0753-4059-8a03-e60bf2c3913b\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " Nov 03 21:04:49.423163 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423120 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-envoy\") pod \"581b6db0-0753-4059-8a03-e60bf2c3913b\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " Nov 03 21:04:49.423163 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423148 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-token\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-token\") pod \"581b6db0-0753-4059-8a03-e60bf2c3913b\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " Nov 03 21:04:49.423597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423176 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-credential-socket\") pod \"581b6db0-0753-4059-8a03-e60bf2c3913b\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " Nov 03 21:04:49.423597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423247 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/581b6db0-0753-4059-8a03-e60bf2c3913b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581b6db0-0753-4059-8a03-e60bf2c3913b-istiod-ca-cert\") pod \"581b6db0-0753-4059-8a03-e60bf2c3913b\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " Nov 03 21:04:49.423597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423284 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-data\") pod \"581b6db0-0753-4059-8a03-e60bf2c3913b\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " Nov 03 21:04:49.423597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423314 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-socket\") pod \"581b6db0-0753-4059-8a03-e60bf2c3913b\" (UID: \"581b6db0-0753-4059-8a03-e60bf2c3913b\") " Nov 03 21:04:49.423597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423321 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-certs" pod "581b6db0-0753-4059-8a03-e60bf2c3913b" (UID: "581b6db0-0753-4059-8a03-e60bf2c3913b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:49.423597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423491 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-credential-socket" pod "581b6db0-0753-4059-8a03-e60bf2c3913b" (UID: "581b6db0-0753-4059-8a03-e60bf2c3913b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:49.423923 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423696 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-data" pod "581b6db0-0753-4059-8a03-e60bf2c3913b" (UID: "581b6db0-0753-4059-8a03-e60bf2c3913b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:49.423923 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423752 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/581b6db0-0753-4059-8a03-e60bf2c3913b-istiod-ca-cert" pod "581b6db0-0753-4059-8a03-e60bf2c3913b" (UID: "581b6db0-0753-4059-8a03-e60bf2c3913b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:49.423923 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423795 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:49.423923 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423817 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:49.423923 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423875 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:49.423923 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.423911 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-socket" pod "581b6db0-0753-4059-8a03-e60bf2c3913b" (UID: "581b6db0-0753-4059-8a03-e60bf2c3913b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:49.427683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.427599 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-envoy" pod "581b6db0-0753-4059-8a03-e60bf2c3913b" (UID: "581b6db0-0753-4059-8a03-e60bf2c3913b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:49.427683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.427628 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-kube-api-access-bz4vb" pod "581b6db0-0753-4059-8a03-e60bf2c3913b" (UID: "581b6db0-0753-4059-8a03-e60bf2c3913b"). InnerVolumeSpecName "kube-api-access-bz4vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:49.427683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.427630 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-podinfo" pod "581b6db0-0753-4059-8a03-e60bf2c3913b" (UID: "581b6db0-0753-4059-8a03-e60bf2c3913b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:04:49.427683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.427652 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-token" pod "581b6db0-0753-4059-8a03-e60bf2c3913b" (UID: "581b6db0-0753-4059-8a03-e60bf2c3913b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:49.525578 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.525535 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz4vb\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-kube-api-access-bz4vb\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:49.525578 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.525576 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:49.525810 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.525590 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:49.525810 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.525606 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/581b6db0-0753-4059-8a03-e60bf2c3913b-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:49.525810 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.525619 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/581b6db0-0753-4059-8a03-e60bf2c3913b-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:49.525810 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.525632 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/581b6db0-0753-4059-8a03-e60bf2c3913b-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:49.725469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.725409 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk"] Nov 03 21:04:49.730911 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.730878 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-421604-openshift-default-6f64b459b5-rckhk"] Nov 03 21:04:49.814144 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:49.814109 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:50.132738 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:50.132702 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:50.261085 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:50.261051 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:50.418002 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:50.417896 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" event={"ID":"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75","Type":"ContainerStarted","Data":"83f2828201c11ff267c8809857474ef08eca394ac82839702c3f8ee74dca73c6"} Nov 03 21:04:50.418002 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:50.417956 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" event={"ID":"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75","Type":"ContainerStarted","Data":"b4a946697ddcce31ad69193b144da511eca4f6824dee44c3287044b83f275820"} Nov 03 21:04:50.420412 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:50.420375 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" event={"ID":"021ae1d1-add4-4c42-a51d-685573728d5a","Type":"ContainerStarted","Data":"3d5722724fd7b2709bcbd1537b4d809c83ad17771442603ab17a008f2ecbe10f"} Nov 03 21:04:50.420557 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:50.420470 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" event={"ID":"021ae1d1-add4-4c42-a51d-685573728d5a","Type":"ContainerStarted","Data":"ed5a5b356b20e72452ec661d30dba41ee6c1f0f2c8fec673891ee15c1b8f2f3b"} Nov 03 21:04:50.443864 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:50.443820 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" podStartSLOduration=2.443805296 podStartE2EDuration="2.443805296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:50.443072763 +0000 UTC m=+3793.893178810" watchObservedRunningTime="2026-11-03 21:04:50.443805296 +0000 UTC m=+3793.893911341" Nov 03 21:04:50.597492 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:50.594089 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:50.724604 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:50.724515 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" path="/var/lib/kubelet/pods/581b6db0-0753-4059-8a03-e60bf2c3913b/volumes" Nov 03 21:04:51.111466 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:51.107591 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:04:51.111466 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:51.107750 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:04:51.111466 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:51.107832 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 21:04:51.115488 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:51.112825 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:04:51.115488 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:51.113374 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 21:04:51.115488 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:51.113434 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 21:04:51.320752 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:51.320702 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:52.108469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:52.096897 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:04:52.108469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:52.096985 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 21:04:52.108469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:52.105713 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:52.121467 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:52.113893 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 21:04:52.121467 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:52.113966 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 21:04:52.734222 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:52.733926 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:52.889469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:52.888847 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:53.020145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:53.020097 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:53.106216 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:53.105799 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:04:53.106216 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:53.105857 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 21:04:53.111931 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:53.111806 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 21:04:53.111931 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:53.111866 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 21:04:53.748837 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:53.748740 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:53.893478 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:53.889837 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:54.030349 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:54.030302 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:54.097196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:54.097161 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:04:54.103472 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:54.097428 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 21:04:54.125339 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:54.125305 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 21:04:54.125935 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:54.125903 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 21:04:54.189538 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:54.189486 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:54.326550 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:54.326274 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:54.595047 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:54.594912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:54.732381 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:54.732343 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:55.104468 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.104351 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Nov 03 21:04:55.104468 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.104408 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Nov 03 21:04:55.118526 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.105285 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:55.118526 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.114323 4711 patch_prober.go:29] interesting pod/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 03 21:04:55.118526 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.114376 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 03 21:04:55.407434 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.407315 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" podStartSLOduration=7.407294458 podStartE2EDuration="7.407294458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:04:50.49754033 +0000 UTC m=+3793.947646375" watchObservedRunningTime="2026-11-03 21:04:55.407294458 +0000 UTC m=+3798.857400507" Nov 03 21:04:55.409645 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.409616 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q"] Nov 03 21:04:55.410095 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.410058 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerName="istio-proxy" containerID="cri-o://83f2828201c11ff267c8809857474ef08eca394ac82839702c3f8ee74dca73c6" gracePeriod=30 Nov 03 21:04:55.412047 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.412024 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69"] Nov 03 21:04:55.412254 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.412232 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" containerName="istio-proxy" containerID="cri-o://3d5722724fd7b2709bcbd1537b4d809c83ad17771442603ab17a008f2ecbe10f" gracePeriod=30 Nov 03 21:04:55.471020 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.470986 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:55.665964 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.665876 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:55.998981 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:55.998876 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:56.818398 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:56.818340 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:57.618293 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:57.618231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:58.313027 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:58.312989 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:59.011639 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:59.011603 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:59.256256 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:59.256215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:59.384846 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:59.384749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:04:59.888664 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:04:59.888627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:00.019847 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:00.019813 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:00.328238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:00.328203 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:00.831319 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:00.831281 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:01.057932 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.057899 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:01.242887 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.242790 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:01.454334 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.454230 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:01.483960 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.483593 4711 generic.go:356] "Generic (PLEG): container finished" podID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerID="83f2828201c11ff267c8809857474ef08eca394ac82839702c3f8ee74dca73c6" exitCode=0 Nov 03 21:05:01.483960 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.483677 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" event={"ID":"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75","Type":"ContainerDied","Data":"83f2828201c11ff267c8809857474ef08eca394ac82839702c3f8ee74dca73c6"} Nov 03 21:05:01.486941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.486788 4711 generic.go:356] "Generic (PLEG): container finished" podID="021ae1d1-add4-4c42-a51d-685573728d5a" containerID="3d5722724fd7b2709bcbd1537b4d809c83ad17771442603ab17a008f2ecbe10f" exitCode=0 Nov 03 21:05:01.486941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.486864 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" event={"ID":"021ae1d1-add4-4c42-a51d-685573728d5a","Type":"ContainerDied","Data":"3d5722724fd7b2709bcbd1537b4d809c83ad17771442603ab17a008f2ecbe10f"} Nov 03 21:05:01.585212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.585179 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:01.594693 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.594668 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:05:01.597625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.597607 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:05:01.694790 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.694749 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-certs\") pod \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " Nov 03 21:05:01.694997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.694796 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-credential-socket\") pod \"021ae1d1-add4-4c42-a51d-685573728d5a\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " Nov 03 21:05:01.694997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.694825 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-token\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-token\") pod \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " Nov 03 21:05:01.694997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.694866 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-certs\") pod \"021ae1d1-add4-4c42-a51d-685573728d5a\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " Nov 03 21:05:01.694997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.694893 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021ae1d1-add4-4c42-a51d-685573728d5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021ae1d1-add4-4c42-a51d-685573728d5a-istiod-ca-cert\") pod \"021ae1d1-add4-4c42-a51d-685573728d5a\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " Nov 03 21:05:01.695190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695037 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/021ae1d1-add4-4c42-a51d-685573728d5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021ae1d1-add4-4c42-a51d-685573728d5a-istio-podinfo\") pod \"021ae1d1-add4-4c42-a51d-685573728d5a\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " Nov 03 21:05:01.695190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695085 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istiod-ca-cert\") pod \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " Nov 03 21:05:01.695190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695045 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-certs" pod "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" (UID: "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.695190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695130 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-kube-api-access-27t6j\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-kube-api-access-27t6j\") pod \"021ae1d1-add4-4c42-a51d-685573728d5a\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " Nov 03 21:05:01.695190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695167 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-credential-socket\") pod \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " Nov 03 21:05:01.695502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695246 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-data\") pod \"021ae1d1-add4-4c42-a51d-685573728d5a\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " Nov 03 21:05:01.695502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695298 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-socket\") pod \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " Nov 03 21:05:01.695502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695327 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-envoy\") pod \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " Nov 03 21:05:01.695502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695371 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-kube-api-access-chv27\") pod \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " Nov 03 21:05:01.695502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695408 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-envoy\") pod \"021ae1d1-add4-4c42-a51d-685573728d5a\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " Nov 03 21:05:01.695502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695481 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-podinfo\") pod \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " Nov 03 21:05:01.695502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695483 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-credential-socket" pod "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" (UID: "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.695885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695502 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istiod-ca-cert" pod "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" (UID: "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:01.695885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695520 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-istio-token\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-istio-token\") pod \"021ae1d1-add4-4c42-a51d-685573728d5a\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " Nov 03 21:05:01.695885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695559 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-socket\") pod \"021ae1d1-add4-4c42-a51d-685573728d5a\" (UID: \"021ae1d1-add4-4c42-a51d-685573728d5a\") " Nov 03 21:05:01.695885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695592 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-data\") pod \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\" (UID: \"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75\") " Nov 03 21:05:01.695885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695622 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-socket" pod "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" (UID: "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.696227 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.695988 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.696227 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.696013 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.696227 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.696028 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.696227 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.696045 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.696227 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.696154 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-data" pod "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" (UID: "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.696681 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.696583 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-credential-socket" pod "021ae1d1-add4-4c42-a51d-685573728d5a" (UID: "021ae1d1-add4-4c42-a51d-685573728d5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.697110 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.696969 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-certs" pod "021ae1d1-add4-4c42-a51d-685573728d5a" (UID: "021ae1d1-add4-4c42-a51d-685573728d5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.697239 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.697196 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-socket" pod "021ae1d1-add4-4c42-a51d-685573728d5a" (UID: "021ae1d1-add4-4c42-a51d-685573728d5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.697239 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.697211 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-data" pod "021ae1d1-add4-4c42-a51d-685573728d5a" (UID: "021ae1d1-add4-4c42-a51d-685573728d5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.697400 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.697367 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021ae1d1-add4-4c42-a51d-685573728d5a-istiod-ca-cert" pod "021ae1d1-add4-4c42-a51d-685573728d5a" (UID: "021ae1d1-add4-4c42-a51d-685573728d5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:01.697940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.697913 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-envoy" pod "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" (UID: "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.698577 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.698545 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-podinfo" pod "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" (UID: "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:01.698696 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.698617 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-token" pod "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" (UID: "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:01.698696 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.698648 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-kube-api-access-chv27" pod "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" (UID: "c1d569bb-dbfc-4f88-95b3-cbf513d3ab75"). InnerVolumeSpecName "kube-api-access-chv27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:01.698838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.698808 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/021ae1d1-add4-4c42-a51d-685573728d5a-istio-podinfo" pod "021ae1d1-add4-4c42-a51d-685573728d5a" (UID: "021ae1d1-add4-4c42-a51d-685573728d5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:01.699151 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.699131 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-envoy" pod "021ae1d1-add4-4c42-a51d-685573728d5a" (UID: "021ae1d1-add4-4c42-a51d-685573728d5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:01.699275 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.699261 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-kube-api-access-27t6j" pod "021ae1d1-add4-4c42-a51d-685573728d5a" (UID: "021ae1d1-add4-4c42-a51d-685573728d5a"). InnerVolumeSpecName "kube-api-access-27t6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:01.699370 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.699351 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-istio-token" pod "021ae1d1-add4-4c42-a51d-685573728d5a" (UID: "021ae1d1-add4-4c42-a51d-685573728d5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:01.796959 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.796921 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.796959 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.796954 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.796959 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.796967 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.796980 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.796993 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.797006 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.797018 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.797029 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.797041 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021ae1d1-add4-4c42-a51d-685573728d5a-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.797062 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021ae1d1-add4-4c42-a51d-685573728d5a-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.797074 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27t6j\" (UniqueName: \"kubernetes.io/projected/021ae1d1-add4-4c42-a51d-685573728d5a-kube-api-access-27t6j\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.797086 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021ae1d1-add4-4c42-a51d-685573728d5a-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.797097 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:01.797229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:01.797109 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chv27\" (UniqueName: \"kubernetes.io/projected/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75-kube-api-access-chv27\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:02.316774 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.316722 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:02.493587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.493559 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" Nov 03 21:05:02.493807 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.493602 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q" event={"ID":"c1d569bb-dbfc-4f88-95b3-cbf513d3ab75","Type":"ContainerDied","Data":"b4a946697ddcce31ad69193b144da511eca4f6824dee44c3287044b83f275820"} Nov 03 21:05:02.493807 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.493678 4711 scope.go:122] "RemoveContainer" containerID="83f2828201c11ff267c8809857474ef08eca394ac82839702c3f8ee74dca73c6" Nov 03 21:05:02.495250 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.495226 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" Nov 03 21:05:02.495392 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.495220 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69" event={"ID":"021ae1d1-add4-4c42-a51d-685573728d5a","Type":"ContainerDied","Data":"ed5a5b356b20e72452ec661d30dba41ee6c1f0f2c8fec673891ee15c1b8f2f3b"} Nov 03 21:05:02.503954 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.503930 4711 scope.go:122] "RemoveContainer" containerID="3d5722724fd7b2709bcbd1537b4d809c83ad17771442603ab17a008f2ecbe10f" Nov 03 21:05:02.518958 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.518930 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q"] Nov 03 21:05:02.523382 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.523357 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-hl29q"] Nov 03 21:05:02.535901 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.535874 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69"] Nov 03 21:05:02.544630 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.544601 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-085566-openshift-default-9958546fc-mvc69"] Nov 03 21:05:02.679377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.679302 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" path="/var/lib/kubelet/pods/021ae1d1-add4-4c42-a51d-685573728d5a/volumes" Nov 03 21:05:02.679784 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:02.679770 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" path="/var/lib/kubelet/pods/c1d569bb-dbfc-4f88-95b3-cbf513d3ab75/volumes" Nov 03 21:05:03.022312 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:03.022275 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:03.790291 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:03.790257 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:04.408230 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:04.408188 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:05.121596 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:05.121555 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:05.815634 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:05.815600 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:06.513681 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:06.513631 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:07.300944 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:07.300893 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:07.436824 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:07.436783 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:08.214127 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:08.214092 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:08.391035 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:08.390990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:08.781780 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:08.781743 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:09.035592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:09.035503 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:09.716440 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:09.716399 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:11.481838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:11.481797 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:11.684060 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:11.684024 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:11.813878 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:11.813825 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:12.524556 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:12.524521 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:12.797077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:12.796972 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:12.939364 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:12.939333 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:13.234687 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:13.234604 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:13.924913 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:13.924854 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:14.626590 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:14.626542 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:14.763057 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:14.763015 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:14.899065 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:14.898967 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:15.339827 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:15.339792 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:15.692069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:15.691974 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:15.826916 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:15.826880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:16.150388 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:16.150343 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:16.654511 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:16.654458 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:17.430868 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:17.430817 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:17.531355 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:05:17.531323 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:05:18.161471 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:18.161406 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:18.931738 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:18.931688 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:19.615743 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:19.615707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:20.384069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:20.384034 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:21.112597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:21.112564 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:21.252797 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:21.252749 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:21.712648 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:21.712613 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:21.919626 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:21.919593 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:22.638363 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:22.638329 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:23.300666 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:23.300636 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:24.113140 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:24.113102 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:24.813327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:24.813293 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:25.514289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:25.514250 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:26.245984 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:26.245937 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:26.918293 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:26.918257 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:27.615244 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:27.615193 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:27.872418 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:27.872325 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc"] Nov 03 21:05:27.970859 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:27.970824 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:28.099995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:28.099951 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:28.555563 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:28.555526 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:29.033730 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:29.033674 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:32.873490 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:32.873407 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" podUID="3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b" containerName="payload-pre-processing" containerID="cri-o://f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891" gracePeriod=25 Nov 03 21:05:33.127496 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.127349 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 21:05:33.213439 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.213404 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-plugins-config-volume\") pod \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\" (UID: \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\") " Nov 03 21:05:33.213630 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.213563 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-kube-api-access-929tm\" (UniqueName: \"kubernetes.io/projected/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-kube-api-access-929tm\") pod \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\" (UID: \"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b\") " Nov 03 21:05:33.213840 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.213812 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-plugins-config-volume" pod "3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b" (UID: "3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:33.215584 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.215565 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-kube-api-access-929tm" pod "3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b" (UID: "3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b"). InnerVolumeSpecName "kube-api-access-929tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:33.314731 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.314700 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:33.314731 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.314726 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-929tm\" (UniqueName: \"kubernetes.io/projected/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b-kube-api-access-929tm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:33.658554 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.658522 4711 generic.go:356] "Generic (PLEG): container finished" podID="3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b" containerID="f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891" exitCode=0 Nov 03 21:05:33.658718 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.658562 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" Nov 03 21:05:33.658718 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.658606 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" event={"ID":"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b","Type":"ContainerDied","Data":"f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891"} Nov 03 21:05:33.658718 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.658656 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc" event={"ID":"3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b","Type":"ContainerDied","Data":"252232ae6ad5a402dfe305b98d69aaa917a853908dbb104ee23c2f35b9f65f6f"} Nov 03 21:05:33.658718 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.658678 4711 scope.go:122] "RemoveContainer" containerID="f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891" Nov 03 21:05:33.668089 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.668068 4711 scope.go:122] "RemoveContainer" containerID="f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891" Nov 03 21:05:33.668359 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:05:33.668339 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891\": container with ID starting with f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891 not found: ID does not exist" containerID="f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891" Nov 03 21:05:33.668429 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.668372 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891"} err="failed to get container status \"f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891\": rpc error: code = NotFound desc = could not find container \"f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891\": container with ID starting with f416bea0081b0614f0f6b09cf5bf781e766e6769b3579b096f29635af9d49891 not found: ID does not exist" Nov 03 21:05:33.681041 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.681020 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc"] Nov 03 21:05:33.685700 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:33.685683 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f3631f-56fd5dd755-l86cc"] Nov 03 21:05:34.677847 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:34.677815 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b" path="/var/lib/kubelet/pods/3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b/volumes" Nov 03 21:05:34.719316 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:34.719286 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:35.843868 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:35.843824 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:35.974991 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:35.974950 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:39.071899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.071865 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv"] Nov 03 21:05:39.072361 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.072102 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" podUID="7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" containerName="maas-api" containerID="cri-o://8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9" gracePeriod=30 Nov 03 21:05:39.157831 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.157798 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:39.339232 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.339205 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 21:05:39.473648 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.473618 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-maas-api-tls\") pod \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " Nov 03 21:05:39.473827 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.473661 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-metrics-certs\") pod \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " Nov 03 21:05:39.473827 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.473720 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-kube-api-access-dwgcg\" (UniqueName: \"kubernetes.io/projected/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-kube-api-access-dwgcg\") pod \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " Nov 03 21:05:39.473827 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.473774 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-openshift-service-ca\") pod \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\" (UID: \"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08\") " Nov 03 21:05:39.474175 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.474146 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-openshift-service-ca" pod "7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" (UID: "7e9bf0e3-5e74-45af-90d6-0f0997cfcd08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:39.475754 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.475729 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-metrics-certs" pod "7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" (UID: "7e9bf0e3-5e74-45af-90d6-0f0997cfcd08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:39.475866 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.475845 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-kube-api-access-dwgcg" pod "7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" (UID: "7e9bf0e3-5e74-45af-90d6-0f0997cfcd08"). InnerVolumeSpecName "kube-api-access-dwgcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:39.475939 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.475858 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-maas-api-tls" pod "7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" (UID: "7e9bf0e3-5e74-45af-90d6-0f0997cfcd08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:39.575552 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.575486 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:39.575552 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.575509 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:39.575552 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.575519 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwgcg\" (UniqueName: \"kubernetes.io/projected/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-kube-api-access-dwgcg\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:39.575552 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.575529 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:39.687676 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.687643 4711 generic.go:356] "Generic (PLEG): container finished" podID="7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" containerID="8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9" exitCode=0 Nov 03 21:05:39.687838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.687690 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" Nov 03 21:05:39.687838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.687720 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" event={"ID":"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08","Type":"ContainerDied","Data":"8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9"} Nov 03 21:05:39.687838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.687767 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv" event={"ID":"7e9bf0e3-5e74-45af-90d6-0f0997cfcd08","Type":"ContainerDied","Data":"470d4a5af8fd3f0e1764807feaf3cf6ef11963c9f4ff99887a17be350255bc98"} Nov 03 21:05:39.687838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.687788 4711 scope.go:122] "RemoveContainer" containerID="8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9" Nov 03 21:05:39.699042 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.699025 4711 scope.go:122] "RemoveContainer" containerID="8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9" Nov 03 21:05:39.699317 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:05:39.699296 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9\": container with ID starting with 8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9 not found: ID does not exist" containerID="8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9" Nov 03 21:05:39.699402 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.699324 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9"} err="failed to get container status \"8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9\": rpc error: code = NotFound desc = could not find container \"8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9\": container with ID starting with 8d1c16696ed4a1a99626a33e82e0e34278e1d1d2c67566971a32116ee34efca9 not found: ID does not exist" Nov 03 21:05:39.712754 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.712721 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv"] Nov 03 21:05:39.716830 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:39.716809 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f3631f-846bcf997d-h2jhv"] Nov 03 21:05:40.678916 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:40.678882 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" path="/var/lib/kubelet/pods/7e9bf0e3-5e74-45af-90d6-0f0997cfcd08/volumes" Nov 03 21:05:40.712962 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:40.712927 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:40.843522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:40.843485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:47.530082 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:05:47.530044 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:05:49.371523 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371475 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk"] Nov 03 21:05:49.371895 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371887 4711 cpu_manager.go:414] "containerMap: removing container" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" Nov 03 21:05:49.371953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371899 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" Nov 03 21:05:49.371953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371925 4711 cpu_manager.go:414] "containerMap: removing container" podUID="3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b" containerName="payload-pre-processing" Nov 03 21:05:49.371953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371930 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b" containerName="payload-pre-processing" Nov 03 21:05:49.371953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371941 4711 cpu_manager.go:414] "containerMap: removing container" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" containerName="istio-proxy" Nov 03 21:05:49.371953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371947 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" containerName="istio-proxy" Nov 03 21:05:49.372111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371957 4711 cpu_manager.go:414] "containerMap: removing container" podUID="7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" containerName="maas-api" Nov 03 21:05:49.372111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371962 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" containerName="maas-api" Nov 03 21:05:49.372111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371969 4711 cpu_manager.go:414] "containerMap: removing container" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerName="istio-proxy" Nov 03 21:05:49.372111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.371973 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerName="istio-proxy" Nov 03 21:05:49.372111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.372053 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d569bb-dbfc-4f88-95b3-cbf513d3ab75" containerName="istio-proxy" Nov 03 21:05:49.372111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.372065 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="581b6db0-0753-4059-8a03-e60bf2c3913b" containerName="istio-proxy" Nov 03 21:05:49.372111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.372074 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="3338f3e3-2bda-4b6b-8fa5-ae3e4fb82d9b" containerName="payload-pre-processing" Nov 03 21:05:49.372111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.372081 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="021ae1d1-add4-4c42-a51d-685573728d5a" containerName="istio-proxy" Nov 03 21:05:49.372111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.372088 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9bf0e3-5e74-45af-90d6-0f0997cfcd08" containerName="maas-api" Nov 03 21:05:49.375202 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.375179 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.379421 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.379398 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3631f-openshift-default-dockercfg-2vnzz\"" Nov 03 21:05:49.387855 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.387831 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk"] Nov 03 21:05:49.473511 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.473474 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.473662 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.473563 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvpr8\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-kube-api-access-gvpr8\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.473662 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.473603 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.473805 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.473664 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.473805 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.473720 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.473805 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.473755 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.473974 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.473845 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.473974 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.473878 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.473974 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.473958 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.574862 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.574836 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.574878 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvpr8\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-kube-api-access-gvpr8\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.574899 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.574914 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.574929 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575264 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.575041 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575264 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.575114 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575366 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.575276 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575366 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.575336 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575366 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.575346 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-credential-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575553 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.575411 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-data\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575553 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.575433 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-socket\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.575647 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-certs\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.575817 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.575792 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istiod-ca-cert\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.577274 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.577247 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-envoy\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.577677 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.577656 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-podinfo\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.587481 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.587433 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-token\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.587687 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.587668 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvpr8\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-kube-api-access-gvpr8\") pod \"e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.690785 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.690728 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:49.971786 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.971757 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk"] Nov 03 21:05:49.973663 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:05:49.973636 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9de9af_5f87_4624_8f2f_7f3e1a80ee3f.slice/crio-aa0797c5ec96f895844818ce7065eaa1d22513812486490b63ec6493d5b98690 WatchSource:0}: Error finding container aa0797c5ec96f895844818ce7065eaa1d22513812486490b63ec6493d5b98690: Status 404 returned error can't find the container with id aa0797c5ec96f895844818ce7065eaa1d22513812486490b63ec6493d5b98690 Nov 03 21:05:49.976027 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.975998 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:05:49.976117 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.976058 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:05:49.976117 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:49.976088 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:05:50.740765 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:50.740721 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" event={"ID":"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f","Type":"ContainerStarted","Data":"8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227"} Nov 03 21:05:50.741217 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:50.740780 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" event={"ID":"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f","Type":"ContainerStarted","Data":"aa0797c5ec96f895844818ce7065eaa1d22513812486490b63ec6493d5b98690"} Nov 03 21:05:50.778153 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:50.778090 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" podStartSLOduration=1.778069707 podStartE2EDuration="1.778069707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:50.764432912 +0000 UTC m=+3854.214538981" watchObservedRunningTime="2026-11-03 21:05:50.778069707 +0000 UTC m=+3854.228175754" Nov 03 21:05:51.585478 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:51.585313 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:51.691636 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:51.691599 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:51.693478 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:51.693440 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:05:51.693609 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:51.693501 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 21:05:51.754694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:51.754665 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:52.692156 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:52.692118 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:05:52.692384 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:52.692198 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 21:05:53.691851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:53.691817 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:05:53.692238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:53.691882 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 21:05:54.691877 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:54.691841 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:05:54.692555 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:54.692513 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 21:05:55.691673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:55.691636 4711 patch_prober.go:29] interesting pod/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 21:05:55.691854 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:55.691686 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 21:05:56.695091 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:56.695057 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:56.695522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:56.695329 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:56.696027 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:56.696001 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:05:57.827200 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:57.827155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:57.957955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:57.957912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:05:59.484262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:05:59.484211 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:00.027470 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:00.027417 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:00.701282 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:00.701241 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:00.954171 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:00.954074 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk"] Nov 03 21:06:00.956430 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:00.954405 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerName="istio-proxy" containerID="cri-o://8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227" gracePeriod=30 Nov 03 21:06:01.034866 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:01.034833 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:01.303607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:01.303574 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:04.324271 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:04.324239 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-69ff9b4cb6-z65tg_81a76879-bf5a-4088-a863-5e5ec0b7213b/manager/0.log" Nov 03 21:06:04.723058 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:04.722985 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ncvvt_de0a63b3-de1f-4ead-9e75-44a17a440fc8/manager/0.log" Nov 03 21:06:05.158684 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:05.158652 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:06.208588 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.208560 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:06:06.255292 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255268 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-socket\") pod \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " Nov 03 21:06:06.255474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255303 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istiod-ca-cert\") pod \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " Nov 03 21:06:06.255474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255363 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-kube-api-access-gvpr8\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-kube-api-access-gvpr8\") pod \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " Nov 03 21:06:06.255474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255386 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-certs\") pod \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " Nov 03 21:06:06.255474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255416 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-data\") pod \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " Nov 03 21:06:06.255474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255472 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-podinfo\") pod \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " Nov 03 21:06:06.255843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255548 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-envoy\") pod \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " Nov 03 21:06:06.255843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255711 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-certs" pod "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" (UID: "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.255843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255752 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-socket" pod "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" (UID: "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.255843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255808 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-credential-socket\") pod \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " Nov 03 21:06:06.256044 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255875 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-token\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-token\") pod \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\" (UID: \"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f\") " Nov 03 21:06:06.256097 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.255813 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istiod-ca-cert" pod "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" (UID: "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:06.256097 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.256039 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-data" pod "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" (UID: "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.256343 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.256322 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.256431 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.256348 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.256431 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.256366 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.256431 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.256379 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.256676 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.256655 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-credential-socket" pod "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" (UID: "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.258037 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.257999 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-envoy" pod "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" (UID: "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:06.258288 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.258261 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-kube-api-access-gvpr8" pod "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" (UID: "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f"). InnerVolumeSpecName "kube-api-access-gvpr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:06.258483 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.258461 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-token" pod "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" (UID: "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:06.258662 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.258639 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-podinfo" pod "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" (UID: "1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:06:06.356919 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.356885 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.356919 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.356912 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.356919 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.356924 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvpr8\" (UniqueName: \"kubernetes.io/projected/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-kube-api-access-gvpr8\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.357191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.356932 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.357191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.356941 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.784949 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.784911 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:06.816885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.816856 4711 generic.go:356] "Generic (PLEG): container finished" podID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerID="8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227" exitCode=0 Nov 03 21:06:06.817028 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.816903 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" Nov 03 21:06:06.817028 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.816942 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" event={"ID":"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f","Type":"ContainerDied","Data":"8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227"} Nov 03 21:06:06.817028 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.817000 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk" event={"ID":"1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f","Type":"ContainerDied","Data":"aa0797c5ec96f895844818ce7065eaa1d22513812486490b63ec6493d5b98690"} Nov 03 21:06:06.817028 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.817023 4711 scope.go:122] "RemoveContainer" containerID="8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227" Nov 03 21:06:06.825756 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.825730 4711 scope.go:122] "RemoveContainer" containerID="8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227" Nov 03 21:06:06.826031 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:06:06.826013 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227\": container with ID starting with 8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227 not found: ID does not exist" containerID="8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227" Nov 03 21:06:06.826119 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.826047 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227"} err="failed to get container status \"8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227\": rpc error: code = NotFound desc = could not find container \"8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227\": container with ID starting with 8a44e00643e177ef5a9cc015ff069cd20e8b8f19b128765863d4f6c871259227 not found: ID does not exist" Nov 03 21:06:06.837383 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.837360 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk"] Nov 03 21:06:06.841611 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:06.841592 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3631f-openshift-default-c7bbc899-wh2nk"] Nov 03 21:06:08.679014 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:08.678984 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" path="/var/lib/kubelet/pods/1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f/volumes" Nov 03 21:06:10.361367 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:10.361330 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:10.923550 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:10.923503 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:11.057127 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:11.057094 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:13.628873 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:13.628820 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:13.757866 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:13.757830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:14.832843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.832809 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:14.906272 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.906233 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8"] Nov 03 21:06:14.906666 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.906653 4711 cpu_manager.go:414] "containerMap: removing container" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerName="istio-proxy" Nov 03 21:06:14.906715 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.906667 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerName="istio-proxy" Nov 03 21:06:14.906769 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.906760 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9de9af-5f87-4624-8f2f-7f3e1a80ee3f" containerName="istio-proxy" Nov 03 21:06:14.909868 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.909852 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:14.917592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.917567 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4deb62c4-dockercfg-6qd4t\"" Nov 03 21:06:14.918139 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.918118 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4deb62c4\"" Nov 03 21:06:14.920069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.920046 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8"] Nov 03 21:06:14.977261 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:14.977201 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:15.042035 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.041988 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0df49c-86a0-4afc-99bc-fff89f45b679-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8\" (UID: \"1f0df49c-86a0-4afc-99bc-fff89f45b679\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:15.042035 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.042042 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285jv\" (UniqueName: \"kubernetes.io/projected/1f0df49c-86a0-4afc-99bc-fff89f45b679-kube-api-access-285jv\") pod \"payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8\" (UID: \"1f0df49c-86a0-4afc-99bc-fff89f45b679\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:15.143353 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.143269 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0df49c-86a0-4afc-99bc-fff89f45b679-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8\" (UID: \"1f0df49c-86a0-4afc-99bc-fff89f45b679\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:15.143353 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.143309 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-285jv\" (UniqueName: \"kubernetes.io/projected/1f0df49c-86a0-4afc-99bc-fff89f45b679-kube-api-access-285jv\") pod \"payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8\" (UID: \"1f0df49c-86a0-4afc-99bc-fff89f45b679\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:15.143969 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.143950 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0df49c-86a0-4afc-99bc-fff89f45b679-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8\" (UID: \"1f0df49c-86a0-4afc-99bc-fff89f45b679\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:15.155789 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.155759 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-285jv\" (UniqueName: \"kubernetes.io/projected/1f0df49c-86a0-4afc-99bc-fff89f45b679-kube-api-access-285jv\") pod \"payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8\" (UID: \"1f0df49c-86a0-4afc-99bc-fff89f45b679\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:15.223697 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.223670 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:15.501543 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.501514 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8"] Nov 03 21:06:15.504042 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:06:15.504019 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0df49c_86a0_4afc_99bc_fff89f45b679.slice/crio-8b04c2cd9a7e9cc51fb2e265cd79398c040ae6926ba8047f4d3556d8c4ec88c1 WatchSource:0}: Error finding container 8b04c2cd9a7e9cc51fb2e265cd79398c040ae6926ba8047f4d3556d8c4ec88c1: Status 404 returned error can't find the container with id 8b04c2cd9a7e9cc51fb2e265cd79398c040ae6926ba8047f4d3556d8c4ec88c1 Nov 03 21:06:15.700018 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.699946 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:15.871514 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.871475 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" event={"ID":"1f0df49c-86a0-4afc-99bc-fff89f45b679","Type":"ContainerStarted","Data":"c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece"} Nov 03 21:06:15.871961 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.871632 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:15.871961 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.871652 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" event={"ID":"1f0df49c-86a0-4afc-99bc-fff89f45b679","Type":"ContainerStarted","Data":"8b04c2cd9a7e9cc51fb2e265cd79398c040ae6926ba8047f4d3556d8c4ec88c1"} Nov 03 21:06:15.890680 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:15.890638 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" podStartSLOduration=1.890623905 podStartE2EDuration="1.890623905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:15.890346937 +0000 UTC m=+3879.340452982" watchObservedRunningTime="2026-11-03 21:06:15.890623905 +0000 UTC m=+3879.340729954" Nov 03 21:06:16.602815 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:16.602729 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:17.323742 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:17.323691 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:17.530379 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:06:17.530342 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:06:18.134545 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:18.134486 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:18.910710 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:18.910670 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:19.732518 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:19.732486 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:20.604974 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:20.604935 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:21.418574 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:21.418539 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:22.167946 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:22.167910 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:22.937903 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:22.937863 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:23.708031 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:23.707994 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:24.518525 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:24.518487 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:25.281435 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:25.281400 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:26.116328 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:26.116291 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:26.815369 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:26.815286 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:26.879849 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:26.879815 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:27.597214 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:27.597170 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:28.437649 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:28.437609 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:29.328519 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.328481 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:29.619412 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.619335 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm"] Nov 03 21:06:29.623274 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.623257 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:29.634224 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.634197 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm"] Nov 03 21:06:29.639979 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.639953 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:06:29.641276 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.641248 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:06:29.641397 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.641312 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:06:29.641397 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.641365 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-48ql9\"" Nov 03 21:06:29.689146 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.689120 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:29.689313 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.689234 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76srf\" (UniqueName: \"kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:29.689411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.689358 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:29.790157 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.790127 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76srf\" (UniqueName: \"kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:29.790289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.790193 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:29.790289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.790229 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:29.790830 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.790806 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:29.792503 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:29.792481 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:29.796956 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:06:29.796937 4711 projected.go:196] Error preparing data for projected volume kube-api-access-76srf for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 21:06:29.797063 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:06:29.797018 4711 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf podName:bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd nodeName:}" failed. No retries permitted until 2026-11-03 21:06:30.296999678 +0000 UTC m=+3893.747105719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-76srf" (UniqueName: "kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf") pod "maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" (UID: "bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 21:06:30.342161 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:30.342122 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:30.397737 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:30.397705 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76srf\" (UniqueName: \"kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:30.409606 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:30.409574 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76srf\" (UniqueName: \"kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf\") pod \"maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:30.538304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:30.538267 4711 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" secret="" err="secret \"maas-api-cleanup-dockercfg-48ql9\" not found" Nov 03 21:06:30.538488 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:30.538321 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:30.810576 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:30.810551 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm"] Nov 03 21:06:30.812350 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:06:30.812325 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccf5c55_b0b1_4f2d_bb93_fe8c0ca613dd.slice/crio-4620f4c229cd8ede28096a81be71b48ab1728ab1a664442fd45c9185243e31dd WatchSource:0}: Error finding container 4620f4c229cd8ede28096a81be71b48ab1728ab1a664442fd45c9185243e31dd: Status 404 returned error can't find the container with id 4620f4c229cd8ede28096a81be71b48ab1728ab1a664442fd45c9185243e31dd Nov 03 21:06:30.959439 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:30.959409 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" event={"ID":"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd","Type":"ContainerStarted","Data":"4620f4c229cd8ede28096a81be71b48ab1728ab1a664442fd45c9185243e31dd"} Nov 03 21:06:31.232893 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:31.232852 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:31.965674 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:31.965644 4711 generic.go:356] "Generic (PLEG): container finished" podID="bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd" containerID="7a90cd80312553c20dab528e7309217d359a3c021e43b6dd0555c2aa29815f1a" exitCode=0 Nov 03 21:06:31.966079 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:31.965730 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" event={"ID":"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd","Type":"ContainerDied","Data":"7a90cd80312553c20dab528e7309217d359a3c021e43b6dd0555c2aa29815f1a"} Nov 03 21:06:32.023192 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:32.023160 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:32.814968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:32.814931 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:33.105708 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.105683 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:33.226657 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.226624 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf\" (UniqueName: \"kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf\") pod \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " Nov 03 21:06:33.226823 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.226674 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-openshift-ca\") pod \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " Nov 03 21:06:33.226873 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.226825 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-xks-ca\") pod \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\" (UID: \"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd\") " Nov 03 21:06:33.227065 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.227042 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-openshift-ca" pod "bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd" (UID: "bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:33.228778 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.228754 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-xks-ca" pod "bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd" (UID: "bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:06:33.228849 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.228777 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf" pod "bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd" (UID: "bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd"). InnerVolumeSpecName "kube-api-access-76srf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:33.275521 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.275492 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh"] Nov 03 21:06:33.275688 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.275668 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" containerID="cri-o://e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e" gracePeriod=30 Nov 03 21:06:33.327566 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.327535 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:33.327566 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.327561 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76srf\" (UniqueName: \"kubernetes.io/projected/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-kube-api-access-76srf\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:33.327566 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.327571 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:33.345383 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.345302 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:33.478732 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.478698 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:33.614794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.614706 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:33.976997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.976918 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" Nov 03 21:06:33.977151 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.976918 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm" event={"ID":"bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd","Type":"ContainerDied","Data":"4620f4c229cd8ede28096a81be71b48ab1728ab1a664442fd45c9185243e31dd"} Nov 03 21:06:33.977151 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:33.977053 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4620f4c229cd8ede28096a81be71b48ab1728ab1a664442fd45c9185243e31dd" Nov 03 21:06:34.205557 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:34.205526 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:34.907505 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:34.907437 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:35.710909 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:35.710811 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:36.516477 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:36.516427 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:37.234337 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:37.234300 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:37.883738 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:37.883713 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:06:37.886678 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:37.886661 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:06:37.931196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:37.931168 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:06:37.934107 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:37.934086 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:06:38.012111 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.012075 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:38.534068 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.534040 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 21:06:38.574593 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574508 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-socket\") pod \"8dda28c4-26f0-4178-8201-b2cb942ff910\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " Nov 03 21:06:38.574593 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574570 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dda28c4-26f0-4178-8201-b2cb942ff910-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dda28c4-26f0-4178-8201-b2cb942ff910-istiod-ca-cert\") pod \"8dda28c4-26f0-4178-8201-b2cb942ff910\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " Nov 03 21:06:38.574842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574629 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-kube-api-access-8hdrd\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-kube-api-access-8hdrd\") pod \"8dda28c4-26f0-4178-8201-b2cb942ff910\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " Nov 03 21:06:38.574842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574718 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-data\") pod \"8dda28c4-26f0-4178-8201-b2cb942ff910\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " Nov 03 21:06:38.574842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574765 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-token\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-token\") pod \"8dda28c4-26f0-4178-8201-b2cb942ff910\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " Nov 03 21:06:38.574842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574769 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-socket" pod "8dda28c4-26f0-4178-8201-b2cb942ff910" (UID: "8dda28c4-26f0-4178-8201-b2cb942ff910"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:38.574842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574812 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-podinfo\") pod \"8dda28c4-26f0-4178-8201-b2cb942ff910\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " Nov 03 21:06:38.575124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574847 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-certs\") pod \"8dda28c4-26f0-4178-8201-b2cb942ff910\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " Nov 03 21:06:38.575124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574923 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-envoy\") pod \"8dda28c4-26f0-4178-8201-b2cb942ff910\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " Nov 03 21:06:38.575124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574975 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-credential-socket\") pod \"8dda28c4-26f0-4178-8201-b2cb942ff910\" (UID: \"8dda28c4-26f0-4178-8201-b2cb942ff910\") " Nov 03 21:06:38.575124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.574991 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-data" pod "8dda28c4-26f0-4178-8201-b2cb942ff910" (UID: "8dda28c4-26f0-4178-8201-b2cb942ff910"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:38.575124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.575013 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dda28c4-26f0-4178-8201-b2cb942ff910-istiod-ca-cert" pod "8dda28c4-26f0-4178-8201-b2cb942ff910" (UID: "8dda28c4-26f0-4178-8201-b2cb942ff910"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:38.575404 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.575210 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-certs" pod "8dda28c4-26f0-4178-8201-b2cb942ff910" (UID: "8dda28c4-26f0-4178-8201-b2cb942ff910"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:38.575404 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.575350 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:38.575546 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.575405 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dda28c4-26f0-4178-8201-b2cb942ff910-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:38.575546 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.575420 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:38.575546 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.575433 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-credential-socket" pod "8dda28c4-26f0-4178-8201-b2cb942ff910" (UID: "8dda28c4-26f0-4178-8201-b2cb942ff910"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:38.575546 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.575432 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:38.577079 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.577055 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-envoy" pod "8dda28c4-26f0-4178-8201-b2cb942ff910" (UID: "8dda28c4-26f0-4178-8201-b2cb942ff910"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:38.577079 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.577068 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-kube-api-access-8hdrd" pod "8dda28c4-26f0-4178-8201-b2cb942ff910" (UID: "8dda28c4-26f0-4178-8201-b2cb942ff910"). InnerVolumeSpecName "kube-api-access-8hdrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:38.577323 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.577304 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-podinfo" pod "8dda28c4-26f0-4178-8201-b2cb942ff910" (UID: "8dda28c4-26f0-4178-8201-b2cb942ff910"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:06:38.577323 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.577309 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-token" pod "8dda28c4-26f0-4178-8201-b2cb942ff910" (UID: "8dda28c4-26f0-4178-8201-b2cb942ff910"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:38.676052 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.676023 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:38.676052 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.676049 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dda28c4-26f0-4178-8201-b2cb942ff910-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:38.676052 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.676060 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hdrd\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-kube-api-access-8hdrd\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:38.676294 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.676070 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:38.676294 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.676078 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dda28c4-26f0-4178-8201-b2cb942ff910-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:38.719829 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:38.719788 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:39.005852 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.005826 4711 generic.go:356] "Generic (PLEG): container finished" podID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerID="e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e" exitCode=0 Nov 03 21:06:39.006000 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.005870 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" event={"ID":"8dda28c4-26f0-4178-8201-b2cb942ff910","Type":"ContainerDied","Data":"e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e"} Nov 03 21:06:39.006000 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.005878 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" Nov 03 21:06:39.006000 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.005915 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh" event={"ID":"8dda28c4-26f0-4178-8201-b2cb942ff910","Type":"ContainerDied","Data":"eda9615fecd93d30af0917edee6854f33f06aaf3fce09ef86cee655e5480d50c"} Nov 03 21:06:39.006000 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.005937 4711 scope.go:122] "RemoveContainer" containerID="e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e" Nov 03 21:06:39.015036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.015008 4711 scope.go:122] "RemoveContainer" containerID="e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e" Nov 03 21:06:39.015303 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:06:39.015285 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e\": container with ID starting with e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e not found: ID does not exist" containerID="e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e" Nov 03 21:06:39.015359 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.015312 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e"} err="failed to get container status \"e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e\": rpc error: code = NotFound desc = could not find container \"e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e\": container with ID starting with e237e4ed0eb570037b6c89168e1ee4f3e4331317559e7ca9600a57853f5f949e not found: ID does not exist" Nov 03 21:06:39.026368 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.026335 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh"] Nov 03 21:06:39.030781 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.030757 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-947bh"] Nov 03 21:06:39.404874 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.404789 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:39.745407 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.745322 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8"] Nov 03 21:06:39.847063 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:39.847022 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:40.018106 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:40.018072 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:40.206796 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:40.206756 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:40.678386 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:40.678353 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" path="/var/lib/kubelet/pods/8dda28c4-26f0-4178-8201-b2cb942ff910/volumes" Nov 03 21:06:40.732313 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:40.732278 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm"] Nov 03 21:06:40.737138 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:40.737109 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4deb62c4-d48wm"] Nov 03 21:06:40.760713 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:40.760684 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:40.893244 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:40.893211 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:42.678541 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:42.678507 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd" path="/var/lib/kubelet/pods/bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd/volumes" Nov 03 21:06:44.746314 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:44.746254 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" podUID="1f0df49c-86a0-4afc-99bc-fff89f45b679" containerName="payload-pre-processing" containerID="cri-o://c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece" gracePeriod=25 Nov 03 21:06:44.996540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:44.996472 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:45.038716 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.038680 4711 generic.go:356] "Generic (PLEG): container finished" podID="1f0df49c-86a0-4afc-99bc-fff89f45b679" containerID="c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece" exitCode=0 Nov 03 21:06:45.038891 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.038728 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" Nov 03 21:06:45.038891 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.038758 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" event={"ID":"1f0df49c-86a0-4afc-99bc-fff89f45b679","Type":"ContainerDied","Data":"c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece"} Nov 03 21:06:45.038891 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.038857 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8" event={"ID":"1f0df49c-86a0-4afc-99bc-fff89f45b679","Type":"ContainerDied","Data":"8b04c2cd9a7e9cc51fb2e265cd79398c040ae6926ba8047f4d3556d8c4ec88c1"} Nov 03 21:06:45.038891 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.038891 4711 scope.go:122] "RemoveContainer" containerID="c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece" Nov 03 21:06:45.048654 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.048634 4711 scope.go:122] "RemoveContainer" containerID="c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece" Nov 03 21:06:45.048982 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:06:45.048960 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece\": container with ID starting with c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece not found: ID does not exist" containerID="c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece" Nov 03 21:06:45.049066 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.048994 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece"} err="failed to get container status \"c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece\": rpc error: code = NotFound desc = could not find container \"c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece\": container with ID starting with c7dc01b4dd2e4cd5b0c1a6102dce4ea476d8ff541a43f47c36c09caf56e73ece not found: ID does not exist" Nov 03 21:06:45.138382 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.138351 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f0df49c-86a0-4afc-99bc-fff89f45b679-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0df49c-86a0-4afc-99bc-fff89f45b679-plugins-config-volume\") pod \"1f0df49c-86a0-4afc-99bc-fff89f45b679\" (UID: \"1f0df49c-86a0-4afc-99bc-fff89f45b679\") " Nov 03 21:06:45.138790 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.138754 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0df49c-86a0-4afc-99bc-fff89f45b679-kube-api-access-285jv\" (UniqueName: \"kubernetes.io/projected/1f0df49c-86a0-4afc-99bc-fff89f45b679-kube-api-access-285jv\") pod \"1f0df49c-86a0-4afc-99bc-fff89f45b679\" (UID: \"1f0df49c-86a0-4afc-99bc-fff89f45b679\") " Nov 03 21:06:45.138907 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.138811 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0df49c-86a0-4afc-99bc-fff89f45b679-plugins-config-volume" pod "1f0df49c-86a0-4afc-99bc-fff89f45b679" (UID: "1f0df49c-86a0-4afc-99bc-fff89f45b679"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:45.139040 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.139026 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f0df49c-86a0-4afc-99bc-fff89f45b679-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:45.140757 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.140727 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0df49c-86a0-4afc-99bc-fff89f45b679-kube-api-access-285jv" pod "1f0df49c-86a0-4afc-99bc-fff89f45b679" (UID: "1f0df49c-86a0-4afc-99bc-fff89f45b679"). InnerVolumeSpecName "kube-api-access-285jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:45.240513 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.240480 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-285jv\" (UniqueName: \"kubernetes.io/projected/1f0df49c-86a0-4afc-99bc-fff89f45b679-kube-api-access-285jv\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:45.361484 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.361439 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8"] Nov 03 21:06:45.365357 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:45.365328 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4deb62c4-5967b55f9f-gpvg8"] Nov 03 21:06:46.120212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:46.118278 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:06:46.678191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:06:46.678158 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0df49c-86a0-4afc-99bc-fff89f45b679" path="/var/lib/kubelet/pods/1f0df49c-86a0-4afc-99bc-fff89f45b679/volumes" Nov 03 21:06:47.531786 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:06:47.531752 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:07:00.085528 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:00.085483 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:01.345533 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:01.345497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:03.368233 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.368194 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr"] Nov 03 21:07:03.368894 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.368872 4711 cpu_manager.go:414] "containerMap: removing container" podUID="bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd" containerName="revoke-keys" Nov 03 21:07:03.368942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.368897 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd" containerName="revoke-keys" Nov 03 21:07:03.368942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.368936 4711 cpu_manager.go:414] "containerMap: removing container" podUID="1f0df49c-86a0-4afc-99bc-fff89f45b679" containerName="payload-pre-processing" Nov 03 21:07:03.369011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.368944 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0df49c-86a0-4afc-99bc-fff89f45b679" containerName="payload-pre-processing" Nov 03 21:07:03.369011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.368958 4711 cpu_manager.go:414] "containerMap: removing container" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" Nov 03 21:07:03.369011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.368965 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" Nov 03 21:07:03.369113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.369078 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccf5c55-b0b1-4f2d-bb93-fe8c0ca613dd" containerName="revoke-keys" Nov 03 21:07:03.369113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.369096 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0df49c-86a0-4afc-99bc-fff89f45b679" containerName="payload-pre-processing" Nov 03 21:07:03.369113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.369106 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dda28c4-26f0-4178-8201-b2cb942ff910" containerName="istio-proxy" Nov 03 21:07:03.375067 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.375032 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.381425 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.381395 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr"] Nov 03 21:07:03.439497 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.439468 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.439497 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.439501 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.439713 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.439532 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1894d83-96c7-4868-b7bb-0ce763ad9893-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.439713 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.439648 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.439788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.439712 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6w6t\" (UniqueName: \"kubernetes.io/projected/c1894d83-96c7-4868-b7bb-0ce763ad9893-kube-api-access-m6w6t\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.439788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.439744 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.439788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.439767 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.439906 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.439891 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.439943 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.439934 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.540611 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.540574 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6w6t\" (UniqueName: \"kubernetes.io/projected/c1894d83-96c7-4868-b7bb-0ce763ad9893-kube-api-access-m6w6t\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.540611 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.540614 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.540890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.540637 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.540890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.540684 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.540890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.540718 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.540890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.540747 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.540890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.540770 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.540890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.540817 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1894d83-96c7-4868-b7bb-0ce763ad9893-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.540890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.540845 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.541219 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.541069 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-workload-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.541288 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.541270 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-data\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.541356 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.541326 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-credential-socket\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.541403 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.541357 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-workload-certs\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.541913 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.541889 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1894d83-96c7-4868-b7bb-0ce763ad9893-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.543210 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.543193 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-envoy\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.543291 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.543270 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-podinfo\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.553216 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.553192 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1894d83-96c7-4868-b7bb-0ce763ad9893-istio-token\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.553827 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.553805 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6w6t\" (UniqueName: \"kubernetes.io/projected/c1894d83-96c7-4868-b7bb-0ce763ad9893-kube-api-access-m6w6t\") pod \"maas-default-gateway-openshift-default-59f957fdd7-7wgtr\" (UID: \"c1894d83-96c7-4868-b7bb-0ce763ad9893\") " pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.698080 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.698008 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:03.978541 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.978514 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr"] Nov 03 21:07:03.980402 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:07:03.980373 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1894d83_96c7_4868_b7bb_0ce763ad9893.slice/crio-c1f6b68c17441e8f85820027af0b99beddee7902ca7bf6d808adcf623eceeddc WatchSource:0}: Error finding container c1f6b68c17441e8f85820027af0b99beddee7902ca7bf6d808adcf623eceeddc: Status 404 returned error can't find the container with id c1f6b68c17441e8f85820027af0b99beddee7902ca7bf6d808adcf623eceeddc Nov 03 21:07:03.982520 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.982489 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:07:03.982623 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.982546 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:07:03.982623 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:03.982576 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:07:04.143618 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:04.143587 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" event={"ID":"c1894d83-96c7-4868-b7bb-0ce763ad9893","Type":"ContainerStarted","Data":"564841f3bc7f541a0d6d2d8b44a24303690f414d5ea887c948b47d92706e5b0d"} Nov 03 21:07:04.143795 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:04.143638 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" event={"ID":"c1894d83-96c7-4868-b7bb-0ce763ad9893","Type":"ContainerStarted","Data":"c1f6b68c17441e8f85820027af0b99beddee7902ca7bf6d808adcf623eceeddc"} Nov 03 21:07:04.168604 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:04.168555 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podStartSLOduration=1.168541428 podStartE2EDuration="1.168541428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:04.166229816 +0000 UTC m=+3927.616335863" watchObservedRunningTime="2026-11-03 21:07:04.168541428 +0000 UTC m=+3927.618647475" Nov 03 21:07:04.698763 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:04.698731 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:05.700245 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:05.700211 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:07:05.700652 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:05.700273 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 21:07:06.699127 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:06.699089 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:07:06.699331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:06.699145 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 21:07:07.698985 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:07.698948 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:07:07.699491 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:07.699007 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 21:07:08.699179 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:08.699144 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:07:08.699617 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:08.699206 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 21:07:08.992177 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:08.992092 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:09.125535 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.125433 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:09.698560 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.698526 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:07:09.698753 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.698586 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 21:07:09.939347 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.939313 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb"] Nov 03 21:07:09.943285 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.943264 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:09.949015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.948963 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:07:09.949196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.949179 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:07:09.950400 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.950383 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:07:09.952477 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.952456 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:07:09.956267 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:09.956243 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb"] Nov 03 21:07:10.032338 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.032300 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-x4jqb\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.032564 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.032362 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lp98\" (UniqueName: \"kubernetes.io/projected/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-kube-api-access-4lp98\") pod \"maas-api-revoke-sub-85237e45-x4jqb\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.032564 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.032494 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-x4jqb\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.133259 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.133220 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lp98\" (UniqueName: \"kubernetes.io/projected/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-kube-api-access-4lp98\") pod \"maas-api-revoke-sub-85237e45-x4jqb\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.133467 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.133269 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-x4jqb\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.133467 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.133359 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-x4jqb\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.134103 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.134078 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-x4jqb\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.135725 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.135704 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-x4jqb\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.147546 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.147525 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lp98\" (UniqueName: \"kubernetes.io/projected/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-kube-api-access-4lp98\") pod \"maas-api-revoke-sub-85237e45-x4jqb\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.259888 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.259857 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:10.536948 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.536918 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb"] Nov 03 21:07:10.539191 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:07:10.539157 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa4d95b_ee33_46d3_a7fd_3a9ec341e2b4.slice/crio-f80d63c696ed1811d1a1f4631a8b82fe893d7a00e57919b974dd7b03567a6936 WatchSource:0}: Error finding container f80d63c696ed1811d1a1f4631a8b82fe893d7a00e57919b974dd7b03567a6936: Status 404 returned error can't find the container with id f80d63c696ed1811d1a1f4631a8b82fe893d7a00e57919b974dd7b03567a6936 Nov 03 21:07:10.698988 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.698954 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Nov 03 21:07:10.699165 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:10.699031 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Nov 03 21:07:11.185352 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:11.185267 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" event={"ID":"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4","Type":"ContainerStarted","Data":"3b968c410f389e22451267ad729497f9122c41bb7b5af28cdecec0aac467cd82"} Nov 03 21:07:11.185352 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:11.185323 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" event={"ID":"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4","Type":"ContainerStarted","Data":"f80d63c696ed1811d1a1f4631a8b82fe893d7a00e57919b974dd7b03567a6936"} Nov 03 21:07:11.206013 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:11.205968 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" podStartSLOduration=2.205956102 podStartE2EDuration="2.205956102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:11.203430938 +0000 UTC m=+3934.653536986" watchObservedRunningTime="2026-11-03 21:07:11.205956102 +0000 UTC m=+3934.656062147" Nov 03 21:07:12.192373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:12.192334 4711 generic.go:356] "Generic (PLEG): container finished" podID="2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4" containerID="3b968c410f389e22451267ad729497f9122c41bb7b5af28cdecec0aac467cd82" exitCode=0 Nov 03 21:07:12.192932 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:12.192425 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" event={"ID":"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4","Type":"ContainerDied","Data":"3b968c410f389e22451267ad729497f9122c41bb7b5af28cdecec0aac467cd82"} Nov 03 21:07:12.699311 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:12.699277 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:12.699510 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:12.699326 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:13.333517 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.333487 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:13.469430 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.469341 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-kube-api-access-4lp98\" (UniqueName: \"kubernetes.io/projected/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-kube-api-access-4lp98\") pod \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " Nov 03 21:07:13.469430 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.469394 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-openshift-ca\") pod \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " Nov 03 21:07:13.469688 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.469437 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-xks-ca\") pod \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\" (UID: \"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4\") " Nov 03 21:07:13.469842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.469816 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-openshift-ca" pod "2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4" (UID: "2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:13.470005 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.469973 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:13.471475 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.471422 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-kube-api-access-4lp98" pod "2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4" (UID: "2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4"). InnerVolumeSpecName "kube-api-access-4lp98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:13.471625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.471603 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-xks-ca" pod "2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4" (UID: "2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:13.570887 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.570845 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lp98\" (UniqueName: \"kubernetes.io/projected/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-kube-api-access-4lp98\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:13.570887 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.570875 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:13.740691 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.740598 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:13.740691 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:13.740671 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:14.206460 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:14.206413 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" Nov 03 21:07:14.206460 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:14.206427 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb" event={"ID":"2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4","Type":"ContainerDied","Data":"f80d63c696ed1811d1a1f4631a8b82fe893d7a00e57919b974dd7b03567a6936"} Nov 03 21:07:14.206759 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:14.206506 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f80d63c696ed1811d1a1f4631a8b82fe893d7a00e57919b974dd7b03567a6936" Nov 03 21:07:14.687373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:14.687337 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:14.687766 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:14.687645 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:14.688271 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:14.688255 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" Nov 03 21:07:17.530276 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:07:17.530242 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:07:21.401939 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:21.401893 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:23.671500 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:23.671462 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:07:47.561723 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:07:47.561598 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:07:50.797747 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.797706 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6"] Nov 03 21:07:50.798341 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.798320 4711 cpu_manager.go:414] "containerMap: removing container" podUID="2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4" containerName="revoke-keys" Nov 03 21:07:50.798433 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.798343 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4" containerName="revoke-keys" Nov 03 21:07:50.798515 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.798460 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4" containerName="revoke-keys" Nov 03 21:07:50.802251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.802228 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:50.806860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.806826 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:07:50.812960 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.808115 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:07:50.812960 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.809682 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:07:50.812960 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.809684 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:07:50.812960 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.812514 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6"] Nov 03 21:07:50.863540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.863508 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgtr\" (UniqueName: \"kubernetes.io/projected/f68d2914-726a-4816-99bd-cecf235f9ba6-kube-api-access-pvgtr\") pod \"maas-api-revoke-sub-f904bff9-xxvt6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:50.863688 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.863648 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xxvt6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:50.863754 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.863692 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xxvt6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:50.964540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.964506 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xxvt6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:50.964705 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.964559 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xxvt6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:50.964705 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.964589 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgtr\" (UniqueName: \"kubernetes.io/projected/f68d2914-726a-4816-99bd-cecf235f9ba6-kube-api-access-pvgtr\") pod \"maas-api-revoke-sub-f904bff9-xxvt6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:50.965299 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.965274 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-xxvt6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:50.966922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.966897 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-xxvt6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:50.977100 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:50.977076 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgtr\" (UniqueName: \"kubernetes.io/projected/f68d2914-726a-4816-99bd-cecf235f9ba6-kube-api-access-pvgtr\") pod \"maas-api-revoke-sub-f904bff9-xxvt6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:51.121634 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:51.121557 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:51.396840 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:51.396811 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6"] Nov 03 21:07:51.399013 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:07:51.398986 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68d2914_726a_4816_99bd_cecf235f9ba6.slice/crio-879d31ab175c1ba6178f222cd9dacb4e64b82759c4db5df407b0fd5127c9843e WatchSource:0}: Error finding container 879d31ab175c1ba6178f222cd9dacb4e64b82759c4db5df407b0fd5127c9843e: Status 404 returned error can't find the container with id 879d31ab175c1ba6178f222cd9dacb4e64b82759c4db5df407b0fd5127c9843e Nov 03 21:07:51.418146 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:51.418119 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" event={"ID":"f68d2914-726a-4816-99bd-cecf235f9ba6","Type":"ContainerStarted","Data":"879d31ab175c1ba6178f222cd9dacb4e64b82759c4db5df407b0fd5127c9843e"} Nov 03 21:07:52.425161 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:52.425075 4711 generic.go:356] "Generic (PLEG): container finished" podID="f68d2914-726a-4816-99bd-cecf235f9ba6" containerID="265bc6a4e58672e6e5effcb08f91213df396a5ce204d1d159f13eca16a51dc45" exitCode=0 Nov 03 21:07:52.425543 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:52.425166 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" event={"ID":"f68d2914-726a-4816-99bd-cecf235f9ba6","Type":"ContainerDied","Data":"265bc6a4e58672e6e5effcb08f91213df396a5ce204d1d159f13eca16a51dc45"} Nov 03 21:07:53.564697 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.564671 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:07:53.692268 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.692191 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-openshift-ca\") pod \"f68d2914-726a-4816-99bd-cecf235f9ba6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " Nov 03 21:07:53.692268 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.692238 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-xks-ca\") pod \"f68d2914-726a-4816-99bd-cecf235f9ba6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " Nov 03 21:07:53.692523 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.692291 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f68d2914-726a-4816-99bd-cecf235f9ba6-kube-api-access-pvgtr\" (UniqueName: \"kubernetes.io/projected/f68d2914-726a-4816-99bd-cecf235f9ba6-kube-api-access-pvgtr\") pod \"f68d2914-726a-4816-99bd-cecf235f9ba6\" (UID: \"f68d2914-726a-4816-99bd-cecf235f9ba6\") " Nov 03 21:07:53.692666 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.692634 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-openshift-ca" pod "f68d2914-726a-4816-99bd-cecf235f9ba6" (UID: "f68d2914-726a-4816-99bd-cecf235f9ba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:53.694378 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.694349 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-xks-ca" pod "f68d2914-726a-4816-99bd-cecf235f9ba6" (UID: "f68d2914-726a-4816-99bd-cecf235f9ba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:53.694531 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.694374 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f68d2914-726a-4816-99bd-cecf235f9ba6-kube-api-access-pvgtr" pod "f68d2914-726a-4816-99bd-cecf235f9ba6" (UID: "f68d2914-726a-4816-99bd-cecf235f9ba6"). InnerVolumeSpecName "kube-api-access-pvgtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:53.794150 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.794114 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:53.794150 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.794151 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f68d2914-726a-4816-99bd-cecf235f9ba6-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:53.794367 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:53.794169 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvgtr\" (UniqueName: \"kubernetes.io/projected/f68d2914-726a-4816-99bd-cecf235f9ba6-kube-api-access-pvgtr\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:54.436270 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:54.436238 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" event={"ID":"f68d2914-726a-4816-99bd-cecf235f9ba6","Type":"ContainerDied","Data":"879d31ab175c1ba6178f222cd9dacb4e64b82759c4db5df407b0fd5127c9843e"} Nov 03 21:07:54.436456 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:54.436283 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="879d31ab175c1ba6178f222cd9dacb4e64b82759c4db5df407b0fd5127c9843e" Nov 03 21:07:54.436456 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:07:54.436247 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6" Nov 03 21:08:00.136248 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:00.136215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:01.402658 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:01.402617 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:14.707643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.707605 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p"] Nov 03 21:08:14.708303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.708029 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f68d2914-726a-4816-99bd-cecf235f9ba6" containerName="revoke-keys" Nov 03 21:08:14.708303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.708040 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f68d2914-726a-4816-99bd-cecf235f9ba6" containerName="revoke-keys" Nov 03 21:08:14.708303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.708131 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f68d2914-726a-4816-99bd-cecf235f9ba6" containerName="revoke-keys" Nov 03 21:08:14.711316 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.711289 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:14.720530 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.719310 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:08:14.720530 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.719425 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:08:14.720530 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.719504 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:08:14.720530 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.720142 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:08:14.722288 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.722264 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p"] Nov 03 21:08:14.855627 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.855598 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-87d6p\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:14.855841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.855650 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnz8x\" (UniqueName: \"kubernetes.io/projected/4d308aa8-1d73-46ab-9267-867b214ac899-kube-api-access-dnz8x\") pod \"maas-api-revoke-sub-f03629db-87d6p\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:14.855841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.855800 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-87d6p\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:14.957163 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.957119 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-87d6p\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:14.957371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.957189 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnz8x\" (UniqueName: \"kubernetes.io/projected/4d308aa8-1d73-46ab-9267-867b214ac899-kube-api-access-dnz8x\") pod \"maas-api-revoke-sub-f03629db-87d6p\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:14.957371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.957267 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-87d6p\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:14.958012 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.957955 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-87d6p\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:14.959560 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.959539 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-87d6p\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:14.969801 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:14.969781 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnz8x\" (UniqueName: \"kubernetes.io/projected/4d308aa8-1d73-46ab-9267-867b214ac899-kube-api-access-dnz8x\") pod \"maas-api-revoke-sub-f03629db-87d6p\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:15.026301 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:15.026262 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:15.299927 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:15.299895 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p"] Nov 03 21:08:15.301883 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:08:15.301856 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d308aa8_1d73_46ab_9267_867b214ac899.slice/crio-4751bfe21b2df6bcd3a3915988f69750787fb00113cf00cea9fdf037f3e0c7fd WatchSource:0}: Error finding container 4751bfe21b2df6bcd3a3915988f69750787fb00113cf00cea9fdf037f3e0c7fd: Status 404 returned error can't find the container with id 4751bfe21b2df6bcd3a3915988f69750787fb00113cf00cea9fdf037f3e0c7fd Nov 03 21:08:15.569488 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:15.569439 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" event={"ID":"4d308aa8-1d73-46ab-9267-867b214ac899","Type":"ContainerStarted","Data":"4751bfe21b2df6bcd3a3915988f69750787fb00113cf00cea9fdf037f3e0c7fd"} Nov 03 21:08:16.575551 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:16.575516 4711 generic.go:356] "Generic (PLEG): container finished" podID="4d308aa8-1d73-46ab-9267-867b214ac899" containerID="7e589f6fb4d47ce0fb3dadb0f4d9eb20260b136d21afb93892ac622d632ef3f7" exitCode=0 Nov 03 21:08:16.575952 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:16.575607 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" event={"ID":"4d308aa8-1d73-46ab-9267-867b214ac899","Type":"ContainerDied","Data":"7e589f6fb4d47ce0fb3dadb0f4d9eb20260b136d21afb93892ac622d632ef3f7"} Nov 03 21:08:17.530511 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:08:17.530474 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:08:17.715737 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.715712 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:17.785107 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.785023 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-xks-ca\") pod \"4d308aa8-1d73-46ab-9267-867b214ac899\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " Nov 03 21:08:17.785268 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.785141 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-openshift-ca\") pod \"4d308aa8-1d73-46ab-9267-867b214ac899\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " Nov 03 21:08:17.785268 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.785242 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d308aa8-1d73-46ab-9267-867b214ac899-kube-api-access-dnz8x\" (UniqueName: \"kubernetes.io/projected/4d308aa8-1d73-46ab-9267-867b214ac899-kube-api-access-dnz8x\") pod \"4d308aa8-1d73-46ab-9267-867b214ac899\" (UID: \"4d308aa8-1d73-46ab-9267-867b214ac899\") " Nov 03 21:08:17.785523 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.785498 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-openshift-ca" pod "4d308aa8-1d73-46ab-9267-867b214ac899" (UID: "4d308aa8-1d73-46ab-9267-867b214ac899"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:17.785611 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.785600 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:17.787269 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.787250 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-xks-ca" pod "4d308aa8-1d73-46ab-9267-867b214ac899" (UID: "4d308aa8-1d73-46ab-9267-867b214ac899"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:17.787325 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.787261 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d308aa8-1d73-46ab-9267-867b214ac899-kube-api-access-dnz8x" pod "4d308aa8-1d73-46ab-9267-867b214ac899" (UID: "4d308aa8-1d73-46ab-9267-867b214ac899"). InnerVolumeSpecName "kube-api-access-dnz8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:17.886562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.886511 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnz8x\" (UniqueName: \"kubernetes.io/projected/4d308aa8-1d73-46ab-9267-867b214ac899-kube-api-access-dnz8x\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:17.886562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:17.886555 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d308aa8-1d73-46ab-9267-867b214ac899-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:18.586977 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:18.586942 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" event={"ID":"4d308aa8-1d73-46ab-9267-867b214ac899","Type":"ContainerDied","Data":"4751bfe21b2df6bcd3a3915988f69750787fb00113cf00cea9fdf037f3e0c7fd"} Nov 03 21:08:18.587159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:18.586986 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p" Nov 03 21:08:18.587159 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:18.586990 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4751bfe21b2df6bcd3a3915988f69750787fb00113cf00cea9fdf037f3e0c7fd" Nov 03 21:08:30.742663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:30.742605 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:08:30.743205 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:30.742695 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:08:31.465721 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.465687 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw"] Nov 03 21:08:31.466245 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.466223 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4d308aa8-1d73-46ab-9267-867b214ac899" containerName="revoke-keys" Nov 03 21:08:31.466245 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.466243 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d308aa8-1d73-46ab-9267-867b214ac899" containerName="revoke-keys" Nov 03 21:08:31.466400 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.466299 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d308aa8-1d73-46ab-9267-867b214ac899" containerName="revoke-keys" Nov 03 21:08:31.469631 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.469613 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.478654 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.477843 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:08:31.478654 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.477872 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:08:31.478654 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.477931 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:08:31.478654 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.477966 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:08:31.480409 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.480385 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw"] Nov 03 21:08:31.556194 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.556166 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmq5\" (UniqueName: \"kubernetes.io/projected/5548f8bf-2ca9-4517-86d5-0933d3949df1-kube-api-access-szmq5\") pod \"maas-api-revoke-sub-0165d7d2-7hwkw\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.556362 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.556205 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7hwkw\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.556362 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.556236 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7hwkw\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.657723 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.657693 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmq5\" (UniqueName: \"kubernetes.io/projected/5548f8bf-2ca9-4517-86d5-0933d3949df1-kube-api-access-szmq5\") pod \"maas-api-revoke-sub-0165d7d2-7hwkw\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.657867 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.657731 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7hwkw\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.657867 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.657757 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7hwkw\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.658390 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.658356 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7hwkw\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.660095 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.660069 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7hwkw\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.670858 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.670827 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmq5\" (UniqueName: \"kubernetes.io/projected/5548f8bf-2ca9-4517-86d5-0933d3949df1-kube-api-access-szmq5\") pod \"maas-api-revoke-sub-0165d7d2-7hwkw\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:31.784524 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:31.784501 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:32.058205 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:32.058181 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw"] Nov 03 21:08:32.060348 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:08:32.060317 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5548f8bf_2ca9_4517_86d5_0933d3949df1.slice/crio-7a4d43704e18e3d33f5eab391e3951d39667c36d1915241fe4d44dd2bc358821 WatchSource:0}: Error finding container 7a4d43704e18e3d33f5eab391e3951d39667c36d1915241fe4d44dd2bc358821: Status 404 returned error can't find the container with id 7a4d43704e18e3d33f5eab391e3951d39667c36d1915241fe4d44dd2bc358821 Nov 03 21:08:32.662155 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:32.662067 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" event={"ID":"5548f8bf-2ca9-4517-86d5-0933d3949df1","Type":"ContainerStarted","Data":"c351bbf1e7892c372c528b752466676b4b9d2d7817ce2a40f8c3ae9ed610af78"} Nov 03 21:08:32.662155 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:32.662122 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" event={"ID":"5548f8bf-2ca9-4517-86d5-0933d3949df1","Type":"ContainerStarted","Data":"7a4d43704e18e3d33f5eab391e3951d39667c36d1915241fe4d44dd2bc358821"} Nov 03 21:08:32.683164 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:32.683114 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" podStartSLOduration=1.68310162 podStartE2EDuration="1.68310162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:32.68034065 +0000 UTC m=+4016.130446699" watchObservedRunningTime="2026-11-03 21:08:32.68310162 +0000 UTC m=+4016.133207665" Nov 03 21:08:33.668519 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:33.668492 4711 generic.go:356] "Generic (PLEG): container finished" podID="5548f8bf-2ca9-4517-86d5-0933d3949df1" containerID="c351bbf1e7892c372c528b752466676b4b9d2d7817ce2a40f8c3ae9ed610af78" exitCode=0 Nov 03 21:08:33.668916 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:33.668570 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" event={"ID":"5548f8bf-2ca9-4517-86d5-0933d3949df1","Type":"ContainerDied","Data":"c351bbf1e7892c372c528b752466676b4b9d2d7817ce2a40f8c3ae9ed610af78"} Nov 03 21:08:34.810991 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.810971 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:34.890237 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.890203 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-openshift-ca\") pod \"5548f8bf-2ca9-4517-86d5-0933d3949df1\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " Nov 03 21:08:34.890408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.890315 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-xks-ca\") pod \"5548f8bf-2ca9-4517-86d5-0933d3949df1\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " Nov 03 21:08:34.890474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.890415 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5548f8bf-2ca9-4517-86d5-0933d3949df1-kube-api-access-szmq5\" (UniqueName: \"kubernetes.io/projected/5548f8bf-2ca9-4517-86d5-0933d3949df1-kube-api-access-szmq5\") pod \"5548f8bf-2ca9-4517-86d5-0933d3949df1\" (UID: \"5548f8bf-2ca9-4517-86d5-0933d3949df1\") " Nov 03 21:08:34.890609 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.890585 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-openshift-ca" pod "5548f8bf-2ca9-4517-86d5-0933d3949df1" (UID: "5548f8bf-2ca9-4517-86d5-0933d3949df1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:34.890750 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.890732 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:34.892390 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.892363 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-xks-ca" pod "5548f8bf-2ca9-4517-86d5-0933d3949df1" (UID: "5548f8bf-2ca9-4517-86d5-0933d3949df1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:34.892390 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.892383 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5548f8bf-2ca9-4517-86d5-0933d3949df1-kube-api-access-szmq5" pod "5548f8bf-2ca9-4517-86d5-0933d3949df1" (UID: "5548f8bf-2ca9-4517-86d5-0933d3949df1"). InnerVolumeSpecName "kube-api-access-szmq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:34.991418 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.991336 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5548f8bf-2ca9-4517-86d5-0933d3949df1-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:34.991418 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:34.991364 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szmq5\" (UniqueName: \"kubernetes.io/projected/5548f8bf-2ca9-4517-86d5-0933d3949df1-kube-api-access-szmq5\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:35.681342 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:35.681309 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" Nov 03 21:08:35.681593 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:35.681360 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw" event={"ID":"5548f8bf-2ca9-4517-86d5-0933d3949df1","Type":"ContainerDied","Data":"7a4d43704e18e3d33f5eab391e3951d39667c36d1915241fe4d44dd2bc358821"} Nov 03 21:08:35.681593 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:35.681404 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a4d43704e18e3d33f5eab391e3951d39667c36d1915241fe4d44dd2bc358821" Nov 03 21:08:42.403172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.403134 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7"] Nov 03 21:08:42.403697 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.403649 4711 cpu_manager.go:414] "containerMap: removing container" podUID="5548f8bf-2ca9-4517-86d5-0933d3949df1" containerName="revoke-keys" Nov 03 21:08:42.403697 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.403663 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5548f8bf-2ca9-4517-86d5-0933d3949df1" containerName="revoke-keys" Nov 03 21:08:42.403817 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.403780 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="5548f8bf-2ca9-4517-86d5-0933d3949df1" containerName="revoke-keys" Nov 03 21:08:42.406997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.406973 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.412245 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.412220 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:08:42.412432 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.412216 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:08:42.412432 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.412231 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:08:42.412432 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.412274 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:08:42.417213 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.417186 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7"] Nov 03 21:08:42.469144 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.469107 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cvwj7\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.469325 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.469283 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8bbp\" (UniqueName: \"kubernetes.io/projected/5abca83b-537a-407c-8b65-b114dbf9f1fb-kube-api-access-m8bbp\") pod \"maas-api-revoke-sub-6b19338f-cvwj7\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.469427 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.469404 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cvwj7\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.570546 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.570508 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8bbp\" (UniqueName: \"kubernetes.io/projected/5abca83b-537a-407c-8b65-b114dbf9f1fb-kube-api-access-m8bbp\") pod \"maas-api-revoke-sub-6b19338f-cvwj7\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.570716 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.570577 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cvwj7\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.570716 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.570606 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cvwj7\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.571219 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.571189 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cvwj7\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.573044 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.573013 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cvwj7\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.584100 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.584072 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8bbp\" (UniqueName: \"kubernetes.io/projected/5abca83b-537a-407c-8b65-b114dbf9f1fb-kube-api-access-m8bbp\") pod \"maas-api-revoke-sub-6b19338f-cvwj7\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:42.725847 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:42.725760 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:43.003858 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:43.003830 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7"] Nov 03 21:08:43.006015 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:08:43.005983 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abca83b_537a_407c_8b65_b114dbf9f1fb.slice/crio-32921ca51a18a1938c3f73a4a21a1b0668de8bd4e6f1bce8e6511d7fccc20dc4 WatchSource:0}: Error finding container 32921ca51a18a1938c3f73a4a21a1b0668de8bd4e6f1bce8e6511d7fccc20dc4: Status 404 returned error can't find the container with id 32921ca51a18a1938c3f73a4a21a1b0668de8bd4e6f1bce8e6511d7fccc20dc4 Nov 03 21:08:43.720244 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:43.720145 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" event={"ID":"5abca83b-537a-407c-8b65-b114dbf9f1fb","Type":"ContainerStarted","Data":"511125e033299459087456806b26e5bb34188e56347ef15f0323a16747cae511"} Nov 03 21:08:43.720244 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:43.720207 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" event={"ID":"5abca83b-537a-407c-8b65-b114dbf9f1fb","Type":"ContainerStarted","Data":"32921ca51a18a1938c3f73a4a21a1b0668de8bd4e6f1bce8e6511d7fccc20dc4"} Nov 03 21:08:43.739398 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:43.739349 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" podStartSLOduration=1.739338123 podStartE2EDuration="1.739338123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:43.738936851 +0000 UTC m=+4027.189042897" watchObservedRunningTime="2026-11-03 21:08:43.739338123 +0000 UTC m=+4027.189444181" Nov 03 21:08:44.726766 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:44.726732 4711 generic.go:356] "Generic (PLEG): container finished" podID="5abca83b-537a-407c-8b65-b114dbf9f1fb" containerID="511125e033299459087456806b26e5bb34188e56347ef15f0323a16747cae511" exitCode=0 Nov 03 21:08:44.727200 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:44.726817 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" event={"ID":"5abca83b-537a-407c-8b65-b114dbf9f1fb","Type":"ContainerDied","Data":"511125e033299459087456806b26e5bb34188e56347ef15f0323a16747cae511"} Nov 03 21:08:45.858293 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:45.858266 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:46.005303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.005272 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-xks-ca\") pod \"5abca83b-537a-407c-8b65-b114dbf9f1fb\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " Nov 03 21:08:46.005524 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.005362 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abca83b-537a-407c-8b65-b114dbf9f1fb-kube-api-access-m8bbp\" (UniqueName: \"kubernetes.io/projected/5abca83b-537a-407c-8b65-b114dbf9f1fb-kube-api-access-m8bbp\") pod \"5abca83b-537a-407c-8b65-b114dbf9f1fb\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " Nov 03 21:08:46.005524 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.005393 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-openshift-ca\") pod \"5abca83b-537a-407c-8b65-b114dbf9f1fb\" (UID: \"5abca83b-537a-407c-8b65-b114dbf9f1fb\") " Nov 03 21:08:46.005828 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.005799 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-openshift-ca" pod "5abca83b-537a-407c-8b65-b114dbf9f1fb" (UID: "5abca83b-537a-407c-8b65-b114dbf9f1fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:46.007538 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.007511 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abca83b-537a-407c-8b65-b114dbf9f1fb-kube-api-access-m8bbp" pod "5abca83b-537a-407c-8b65-b114dbf9f1fb" (UID: "5abca83b-537a-407c-8b65-b114dbf9f1fb"). InnerVolumeSpecName "kube-api-access-m8bbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:46.007654 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.007635 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-xks-ca" pod "5abca83b-537a-407c-8b65-b114dbf9f1fb" (UID: "5abca83b-537a-407c-8b65-b114dbf9f1fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:46.106968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.106935 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:46.106968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.106965 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8bbp\" (UniqueName: \"kubernetes.io/projected/5abca83b-537a-407c-8b65-b114dbf9f1fb-kube-api-access-m8bbp\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:46.106968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.106975 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5abca83b-537a-407c-8b65-b114dbf9f1fb-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:46.739813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.739776 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" event={"ID":"5abca83b-537a-407c-8b65-b114dbf9f1fb","Type":"ContainerDied","Data":"32921ca51a18a1938c3f73a4a21a1b0668de8bd4e6f1bce8e6511d7fccc20dc4"} Nov 03 21:08:46.739813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.739795 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7" Nov 03 21:08:46.740062 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:46.739823 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32921ca51a18a1938c3f73a4a21a1b0668de8bd4e6f1bce8e6511d7fccc20dc4" Nov 03 21:08:47.530945 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:08:47.530910 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:08:53.943505 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:53.942707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:54.089468 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.089080 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:54.732468 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.722041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:08:54.792384 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.792341 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-657df65587-bkr92"] Nov 03 21:08:54.793355 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.793326 4711 cpu_manager.go:414] "containerMap: removing container" podUID="5abca83b-537a-407c-8b65-b114dbf9f1fb" containerName="revoke-keys" Nov 03 21:08:54.793563 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.793546 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abca83b-537a-407c-8b65-b114dbf9f1fb" containerName="revoke-keys" Nov 03 21:08:54.793836 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.793820 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abca83b-537a-407c-8b65-b114dbf9f1fb" containerName="revoke-keys" Nov 03 21:08:54.858148 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.857776 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:54.858683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.858651 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-657df65587-bkr92"] Nov 03 21:08:54.862323 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.862300 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 21:08:54.863175 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.863156 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bc2b285ae46f\"" Nov 03 21:08:54.863307 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.863206 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 21:08:54.863307 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.863256 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 21:08:54.863307 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.863304 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 21:08:54.863522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:54.863391 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rkltj\"" Nov 03 21:08:55.050965 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.050923 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwjb\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-kube-api-access-qqwjb\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.051503 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.051475 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-provider-credentials\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.051689 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.051676 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-plugins-config-volume\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.051858 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.051837 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-routing-overlay-volume\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.152725 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.152687 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwjb\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-kube-api-access-qqwjb\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.152910 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.152748 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-provider-credentials\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.152910 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.152790 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-plugins-config-volume\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.152910 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.152848 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-routing-overlay-volume\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.153598 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.153573 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-routing-overlay-volume\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.153834 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.153815 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-plugins-config-volume\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.155416 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.155394 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-provider-credentials\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.169469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.165612 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwjb\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-kube-api-access-qqwjb\") pod \"payload-processing-external-model-657df65587-bkr92\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.247730 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.247697 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.634802 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.634663 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-657df65587-bkr92"] Nov 03 21:08:55.863530 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.863477 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" event={"ID":"055e4b09-eb0b-412c-a33c-6f422a03a64c","Type":"ContainerStarted","Data":"8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8"} Nov 03 21:08:55.863721 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.863556 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:08:55.863721 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.863581 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" event={"ID":"055e4b09-eb0b-412c-a33c-6f422a03a64c","Type":"ContainerStarted","Data":"9ea4ce4f9de60943152fa9644521fc71c90e32b262f782affc5f4532155e2fda"} Nov 03 21:08:55.884137 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:08:55.884081 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" podStartSLOduration=1.88406729 podStartE2EDuration="1.88406729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:55.882402963 +0000 UTC m=+4039.332509009" watchObservedRunningTime="2026-11-03 21:08:55.88406729 +0000 UTC m=+4039.334173336" Nov 03 21:09:00.176579 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.176529 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:00.561515 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.561482 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv"] Nov 03 21:09:00.564899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.564876 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.573556 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.573008 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:09:00.573556 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.573083 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:09:00.573556 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.573159 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:09:00.574406 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.573843 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:09:00.575276 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.575247 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv"] Nov 03 21:09:00.654892 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.654859 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ce3d9c-f8snv\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.655077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.654920 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w457\" (UniqueName: \"kubernetes.io/projected/05adac70-8ddc-4f3a-9bb9-138b3bef5193-kube-api-access-5w457\") pod \"maas-api-revoke-sub-36ce3d9c-f8snv\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.655077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.655017 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ce3d9c-f8snv\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.755780 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.755738 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ce3d9c-f8snv\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.755994 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.755971 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w457\" (UniqueName: \"kubernetes.io/projected/05adac70-8ddc-4f3a-9bb9-138b3bef5193-kube-api-access-5w457\") pod \"maas-api-revoke-sub-36ce3d9c-f8snv\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.756080 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.756041 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ce3d9c-f8snv\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.756689 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.756663 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36ce3d9c-f8snv\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.758198 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.758166 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36ce3d9c-f8snv\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.768609 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.768586 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w457\" (UniqueName: \"kubernetes.io/projected/05adac70-8ddc-4f3a-9bb9-138b3bef5193-kube-api-access-5w457\") pod \"maas-api-revoke-sub-36ce3d9c-f8snv\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:00.880641 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:00.880564 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:01.158619 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:01.158586 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv"] Nov 03 21:09:01.160178 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:09:01.160146 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05adac70_8ddc_4f3a_9bb9_138b3bef5193.slice/crio-8114202f840948f5c98127bca2c4a8bf6d2e6381e92f6d4aafe447d434a8296c WatchSource:0}: Error finding container 8114202f840948f5c98127bca2c4a8bf6d2e6381e92f6d4aafe447d434a8296c: Status 404 returned error can't find the container with id 8114202f840948f5c98127bca2c4a8bf6d2e6381e92f6d4aafe447d434a8296c Nov 03 21:09:01.449431 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:01.449340 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:01.932934 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:01.932897 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" event={"ID":"05adac70-8ddc-4f3a-9bb9-138b3bef5193","Type":"ContainerStarted","Data":"f87bbe5a2fc44085daaa31d0688a059b3d1ef99b709423f1a1c55013404e59a0"} Nov 03 21:09:01.933145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:01.932947 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" event={"ID":"05adac70-8ddc-4f3a-9bb9-138b3bef5193","Type":"ContainerStarted","Data":"8114202f840948f5c98127bca2c4a8bf6d2e6381e92f6d4aafe447d434a8296c"} Nov 03 21:09:01.952744 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:01.952691 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" podStartSLOduration=1.952679354 podStartE2EDuration="1.952679354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:01.951008904 +0000 UTC m=+4045.401114951" watchObservedRunningTime="2026-11-03 21:09:01.952679354 +0000 UTC m=+4045.402785400" Nov 03 21:09:02.939375 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:02.939341 4711 generic.go:356] "Generic (PLEG): container finished" podID="05adac70-8ddc-4f3a-9bb9-138b3bef5193" containerID="f87bbe5a2fc44085daaa31d0688a059b3d1ef99b709423f1a1c55013404e59a0" exitCode=0 Nov 03 21:09:02.939820 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:02.939431 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" event={"ID":"05adac70-8ddc-4f3a-9bb9-138b3bef5193","Type":"ContainerDied","Data":"f87bbe5a2fc44085daaa31d0688a059b3d1ef99b709423f1a1c55013404e59a0"} Nov 03 21:09:04.084792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.084765 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:04.193971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.193934 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-openshift-ca\") pod \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " Nov 03 21:09:04.194173 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.194061 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-xks-ca\") pod \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " Nov 03 21:09:04.194173 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.194135 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05adac70-8ddc-4f3a-9bb9-138b3bef5193-kube-api-access-5w457\" (UniqueName: \"kubernetes.io/projected/05adac70-8ddc-4f3a-9bb9-138b3bef5193-kube-api-access-5w457\") pod \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\" (UID: \"05adac70-8ddc-4f3a-9bb9-138b3bef5193\") " Nov 03 21:09:04.194338 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.194314 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-openshift-ca" pod "05adac70-8ddc-4f3a-9bb9-138b3bef5193" (UID: "05adac70-8ddc-4f3a-9bb9-138b3bef5193"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:04.194544 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.194526 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:04.196156 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.196110 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05adac70-8ddc-4f3a-9bb9-138b3bef5193-kube-api-access-5w457" pod "05adac70-8ddc-4f3a-9bb9-138b3bef5193" (UID: "05adac70-8ddc-4f3a-9bb9-138b3bef5193"). InnerVolumeSpecName "kube-api-access-5w457". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:04.196220 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.196162 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-xks-ca" pod "05adac70-8ddc-4f3a-9bb9-138b3bef5193" (UID: "05adac70-8ddc-4f3a-9bb9-138b3bef5193"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:04.296130 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.296095 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w457\" (UniqueName: \"kubernetes.io/projected/05adac70-8ddc-4f3a-9bb9-138b3bef5193-kube-api-access-5w457\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:04.296130 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.296124 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05adac70-8ddc-4f3a-9bb9-138b3bef5193-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:04.951145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.951059 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" event={"ID":"05adac70-8ddc-4f3a-9bb9-138b3bef5193","Type":"ContainerDied","Data":"8114202f840948f5c98127bca2c4a8bf6d2e6381e92f6d4aafe447d434a8296c"} Nov 03 21:09:04.951145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.951090 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv" Nov 03 21:09:04.951145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:04.951117 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8114202f840948f5c98127bca2c4a8bf6d2e6381e92f6d4aafe447d434a8296c" Nov 03 21:09:06.873581 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:06.873551 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:09:11.489224 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:11.489187 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-657df65587-bkr92"] Nov 03 21:09:11.497984 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:11.497938 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:11.633195 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:11.633157 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:15.420676 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.420643 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz"] Nov 03 21:09:15.421146 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.421131 4711 cpu_manager.go:414] "containerMap: removing container" podUID="05adac70-8ddc-4f3a-9bb9-138b3bef5193" containerName="revoke-keys" Nov 03 21:09:15.421189 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.421148 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05adac70-8ddc-4f3a-9bb9-138b3bef5193" containerName="revoke-keys" Nov 03 21:09:15.421277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.421267 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="05adac70-8ddc-4f3a-9bb9-138b3bef5193" containerName="revoke-keys" Nov 03 21:09:15.424371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.424354 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.433506 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.433479 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-0704dddb-kserve-self-signed-certs\"" Nov 03 21:09:15.436495 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.436466 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz"] Nov 03 21:09:15.498702 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.498621 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:15.553086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.553046 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tmp-dir\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.553276 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.553101 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfbk\" (UniqueName: \"kubernetes.io/projected/9b2fef45-8c85-48ce-8575-ede14cb79ce4-kube-api-access-pcfbk\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.553276 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.553191 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tls-certs\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.553276 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.553226 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-dshm\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.553276 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.553254 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-model-cache\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.553502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.553355 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-home\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.654382 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.654347 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tmp-dir\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.654382 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.654383 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfbk\" (UniqueName: \"kubernetes.io/projected/9b2fef45-8c85-48ce-8575-ede14cb79ce4-kube-api-access-pcfbk\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.654663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.654405 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tls-certs\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.654663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.654427 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-dshm\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.654663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.654474 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-model-cache\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.654663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.654502 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-home\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.654847 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.654788 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tmp-dir\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.654911 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.654886 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-model-cache\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.654989 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.654970 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-home\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.656787 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.656752 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-dshm\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.656921 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.656903 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tls-certs\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.667388 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.667366 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfbk\" (UniqueName: \"kubernetes.io/projected/9b2fef45-8c85-48ce-8575-ede14cb79ce4-kube-api-access-pcfbk\") pod \"e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.738371 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.738280 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:15.814693 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:15.814637 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:16.020856 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.020826 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz"] Nov 03 21:09:16.022643 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:09:16.022603 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b2fef45_8c85_48ce_8575_ede14cb79ce4.slice/crio-74ee0514bff58ac8f93fe9a2f3d18f4184471d5bb287d8bcb980c7c1b60e58d9 WatchSource:0}: Error finding container 74ee0514bff58ac8f93fe9a2f3d18f4184471d5bb287d8bcb980c7c1b60e58d9: Status 404 returned error can't find the container with id 74ee0514bff58ac8f93fe9a2f3d18f4184471d5bb287d8bcb980c7c1b60e58d9 Nov 03 21:09:16.069790 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.069763 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" event={"ID":"9b2fef45-8c85-48ce-8575-ede14cb79ce4","Type":"ContainerStarted","Data":"74ee0514bff58ac8f93fe9a2f3d18f4184471d5bb287d8bcb980c7c1b60e58d9"} Nov 03 21:09:16.454392 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.454278 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:16.489936 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.489900 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" podUID="055e4b09-eb0b-412c-a33c-6f422a03a64c" containerName="payload-processing" containerID="cri-o://8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8" gracePeriod=25 Nov 03 21:09:16.743647 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.743617 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:09:16.868079 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.868039 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-plugins-config-volume\") pod \"055e4b09-eb0b-412c-a33c-6f422a03a64c\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " Nov 03 21:09:16.868391 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.868102 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-routing-overlay-volume\") pod \"055e4b09-eb0b-412c-a33c-6f422a03a64c\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " Nov 03 21:09:16.868391 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.868142 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-provider-credentials\") pod \"055e4b09-eb0b-412c-a33c-6f422a03a64c\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " Nov 03 21:09:16.868391 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.868296 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-kube-api-access-qqwjb\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-kube-api-access-qqwjb\") pod \"055e4b09-eb0b-412c-a33c-6f422a03a64c\" (UID: \"055e4b09-eb0b-412c-a33c-6f422a03a64c\") " Nov 03 21:09:16.868580 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.868421 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-routing-overlay-volume" pod "055e4b09-eb0b-412c-a33c-6f422a03a64c" (UID: "055e4b09-eb0b-412c-a33c-6f422a03a64c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:16.868580 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.868519 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-plugins-config-volume" pod "055e4b09-eb0b-412c-a33c-6f422a03a64c" (UID: "055e4b09-eb0b-412c-a33c-6f422a03a64c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:16.868699 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.868652 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:16.868699 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.868665 4711 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/055e4b09-eb0b-412c-a33c-6f422a03a64c-routing-overlay-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:16.870360 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.870331 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-kube-api-access-qqwjb" pod "055e4b09-eb0b-412c-a33c-6f422a03a64c" (UID: "055e4b09-eb0b-412c-a33c-6f422a03a64c"). InnerVolumeSpecName "kube-api-access-qqwjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:16.870360 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.870352 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-provider-credentials" pod "055e4b09-eb0b-412c-a33c-6f422a03a64c" (UID: "055e4b09-eb0b-412c-a33c-6f422a03a64c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:16.970225 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.970144 4711 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-provider-credentials\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:16.970225 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:16.970172 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwjb\" (UniqueName: \"kubernetes.io/projected/055e4b09-eb0b-412c-a33c-6f422a03a64c-kube-api-access-qqwjb\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:17.077086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.077051 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" event={"ID":"9b2fef45-8c85-48ce-8575-ede14cb79ce4","Type":"ContainerStarted","Data":"55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca"} Nov 03 21:09:17.078437 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.078412 4711 generic.go:356] "Generic (PLEG): container finished" podID="055e4b09-eb0b-412c-a33c-6f422a03a64c" containerID="8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8" exitCode=0 Nov 03 21:09:17.078587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.078462 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" event={"ID":"055e4b09-eb0b-412c-a33c-6f422a03a64c","Type":"ContainerDied","Data":"8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8"} Nov 03 21:09:17.078587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.078468 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" Nov 03 21:09:17.078587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.078497 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-657df65587-bkr92" event={"ID":"055e4b09-eb0b-412c-a33c-6f422a03a64c","Type":"ContainerDied","Data":"9ea4ce4f9de60943152fa9644521fc71c90e32b262f782affc5f4532155e2fda"} Nov 03 21:09:17.078587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.078517 4711 scope.go:122] "RemoveContainer" containerID="8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8" Nov 03 21:09:17.088083 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.088067 4711 scope.go:122] "RemoveContainer" containerID="8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8" Nov 03 21:09:17.088351 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:09:17.088328 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8\": container with ID starting with 8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8 not found: ID does not exist" containerID="8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8" Nov 03 21:09:17.088434 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.088354 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8"} err="failed to get container status \"8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8\": rpc error: code = NotFound desc = could not find container \"8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8\": container with ID starting with 8aa5ecfaea036ceef369e781bbe13208c80901e78a7d9ca012dc7d83472dc6a8 not found: ID does not exist" Nov 03 21:09:17.098268 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.098230 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" podStartSLOduration=2.098218519 podStartE2EDuration="2.098218519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:17.09708141 +0000 UTC m=+4060.547187458" watchObservedRunningTime="2026-11-03 21:09:17.098218519 +0000 UTC m=+4060.548324599" Nov 03 21:09:17.117938 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.117905 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-657df65587-bkr92"] Nov 03 21:09:17.122245 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.122223 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-657df65587-bkr92"] Nov 03 21:09:17.532145 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:09:17.532109 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:09:17.738609 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.738576 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:17.751236 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:17.751206 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:18.680117 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:18.680080 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055e4b09-eb0b-412c-a33c-6f422a03a64c" path="/var/lib/kubelet/pods/055e4b09-eb0b-412c-a33c-6f422a03a64c/volumes" Nov 03 21:09:21.918344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:21.918305 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:22.054947 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:22.054883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:09:25.738877 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:25.738836 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:25.751141 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:25.751117 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:26.176579 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:26.176546 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:09:47.530503 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:09:47.530463 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:09:48.017090 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:09:48.017052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:00.253873 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:00.253805 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:01.505273 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:01.505235 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:04.425189 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.425157 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb"] Nov 03 21:10:04.425636 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.425623 4711 cpu_manager.go:414] "containerMap: removing container" podUID="055e4b09-eb0b-412c-a33c-6f422a03a64c" containerName="payload-processing" Nov 03 21:10:04.425683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.425637 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055e4b09-eb0b-412c-a33c-6f422a03a64c" containerName="payload-processing" Nov 03 21:10:04.425728 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.425712 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="055e4b09-eb0b-412c-a33c-6f422a03a64c" containerName="payload-processing" Nov 03 21:10:04.429209 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.429193 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.437210 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.437189 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:10:04.437330 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.437282 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:10:04.437330 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.437309 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:10:04.437738 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.437699 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:10:04.439558 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.439534 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb"] Nov 03 21:10:04.509922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.509885 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d7f08ad-fz5nb\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.509922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.509920 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d7f08ad-fz5nb\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.510201 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.509945 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjxht\" (UniqueName: \"kubernetes.io/projected/3e7b168c-32e6-4726-b339-338cf4250813-kube-api-access-xjxht\") pod \"maas-api-revoke-sub-2d7f08ad-fz5nb\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.611334 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.611304 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d7f08ad-fz5nb\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.611533 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.611341 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d7f08ad-fz5nb\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.611533 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.611375 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjxht\" (UniqueName: \"kubernetes.io/projected/3e7b168c-32e6-4726-b339-338cf4250813-kube-api-access-xjxht\") pod \"maas-api-revoke-sub-2d7f08ad-fz5nb\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.612050 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.612025 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d7f08ad-fz5nb\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.613682 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.613652 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d7f08ad-fz5nb\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.623701 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.623677 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjxht\" (UniqueName: \"kubernetes.io/projected/3e7b168c-32e6-4726-b339-338cf4250813-kube-api-access-xjxht\") pod \"maas-api-revoke-sub-2d7f08ad-fz5nb\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.743741 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.743681 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:04.856262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.856230 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:04.982468 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:04.982412 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:05.017436 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:05.017409 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb"] Nov 03 21:10:05.019407 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:10:05.019383 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7b168c_32e6_4726_b339_338cf4250813.slice/crio-0efc21674bd5d62d7be648db10f7cd9ea02df54f1c8d3b0604e774a8555c50c5 WatchSource:0}: Error finding container 0efc21674bd5d62d7be648db10f7cd9ea02df54f1c8d3b0604e774a8555c50c5: Status 404 returned error can't find the container with id 0efc21674bd5d62d7be648db10f7cd9ea02df54f1c8d3b0604e774a8555c50c5 Nov 03 21:10:05.381232 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:05.381195 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" event={"ID":"3e7b168c-32e6-4726-b339-338cf4250813","Type":"ContainerStarted","Data":"790938efaa3288762b38b3f4c851ef2c3ff81ae3a4c8425d5dcb8a8e74760ad5"} Nov 03 21:10:05.381416 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:05.381261 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" event={"ID":"3e7b168c-32e6-4726-b339-338cf4250813","Type":"ContainerStarted","Data":"0efc21674bd5d62d7be648db10f7cd9ea02df54f1c8d3b0604e774a8555c50c5"} Nov 03 21:10:05.401176 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:05.401129 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" podStartSLOduration=1.401116046 podStartE2EDuration="1.401116046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:05.39910172 +0000 UTC m=+4108.849207766" watchObservedRunningTime="2026-11-03 21:10:05.401116046 +0000 UTC m=+4108.851222092" Nov 03 21:10:06.388537 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:06.388507 4711 generic.go:356] "Generic (PLEG): container finished" podID="3e7b168c-32e6-4726-b339-338cf4250813" containerID="790938efaa3288762b38b3f4c851ef2c3ff81ae3a4c8425d5dcb8a8e74760ad5" exitCode=0 Nov 03 21:10:06.388949 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:06.388599 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" event={"ID":"3e7b168c-32e6-4726-b339-338cf4250813","Type":"ContainerDied","Data":"790938efaa3288762b38b3f4c851ef2c3ff81ae3a4c8425d5dcb8a8e74760ad5"} Nov 03 21:10:07.530660 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.530637 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:07.642633 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.642599 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-xks-ca\") pod \"3e7b168c-32e6-4726-b339-338cf4250813\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " Nov 03 21:10:07.642810 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.642656 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7b168c-32e6-4726-b339-338cf4250813-kube-api-access-xjxht\" (UniqueName: \"kubernetes.io/projected/3e7b168c-32e6-4726-b339-338cf4250813-kube-api-access-xjxht\") pod \"3e7b168c-32e6-4726-b339-338cf4250813\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " Nov 03 21:10:07.642810 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.642768 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-openshift-ca\") pod \"3e7b168c-32e6-4726-b339-338cf4250813\" (UID: \"3e7b168c-32e6-4726-b339-338cf4250813\") " Nov 03 21:10:07.643114 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.643091 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-openshift-ca" pod "3e7b168c-32e6-4726-b339-338cf4250813" (UID: "3e7b168c-32e6-4726-b339-338cf4250813"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:07.644759 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.644700 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-xks-ca" pod "3e7b168c-32e6-4726-b339-338cf4250813" (UID: "3e7b168c-32e6-4726-b339-338cf4250813"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:07.644759 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.644748 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7b168c-32e6-4726-b339-338cf4250813-kube-api-access-xjxht" pod "3e7b168c-32e6-4726-b339-338cf4250813" (UID: "3e7b168c-32e6-4726-b339-338cf4250813"). InnerVolumeSpecName "kube-api-access-xjxht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:07.744142 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.744115 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjxht\" (UniqueName: \"kubernetes.io/projected/3e7b168c-32e6-4726-b339-338cf4250813-kube-api-access-xjxht\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.744281 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.744145 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:07.744281 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:07.744165 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e7b168c-32e6-4726-b339-338cf4250813-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:08.400042 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:08.400012 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" Nov 03 21:10:08.400232 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:08.400012 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb" event={"ID":"3e7b168c-32e6-4726-b339-338cf4250813","Type":"ContainerDied","Data":"0efc21674bd5d62d7be648db10f7cd9ea02df54f1c8d3b0604e774a8555c50c5"} Nov 03 21:10:08.400232 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:08.400134 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0efc21674bd5d62d7be648db10f7cd9ea02df54f1c8d3b0604e774a8555c50c5" Nov 03 21:10:15.880610 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:15.880569 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz"] Nov 03 21:10:15.917668 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:15.917627 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:16.054076 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:16.054042 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:16.901876 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:16.901835 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:17.532630 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:10:17.532585 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:10:20.725881 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.725841 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc"] Nov 03 21:10:20.726512 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.726502 4711 cpu_manager.go:414] "containerMap: removing container" podUID="3e7b168c-32e6-4726-b339-338cf4250813" containerName="revoke-keys" Nov 03 21:10:20.726589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.726518 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7b168c-32e6-4726-b339-338cf4250813" containerName="revoke-keys" Nov 03 21:10:20.726650 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.726636 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7b168c-32e6-4726-b339-338cf4250813" containerName="revoke-keys" Nov 03 21:10:20.730127 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.730104 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:20.737630 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.737606 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:10:20.737778 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.737673 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:10:20.737864 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.737788 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:10:20.737864 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.737855 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:10:20.744985 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.744960 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc"] Nov 03 21:10:20.773107 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.773064 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1b39fb-5dbvc\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:20.773322 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.773217 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1b39fb-5dbvc\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:20.773400 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.773381 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56b92\" (UniqueName: \"kubernetes.io/projected/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-kube-api-access-56b92\") pod \"maas-api-revoke-sub-4e1b39fb-5dbvc\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:20.874710 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.874673 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56b92\" (UniqueName: \"kubernetes.io/projected/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-kube-api-access-56b92\") pod \"maas-api-revoke-sub-4e1b39fb-5dbvc\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:20.874931 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.874752 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1b39fb-5dbvc\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:20.874931 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.874828 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1b39fb-5dbvc\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:20.875673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.875642 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e1b39fb-5dbvc\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:20.877410 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.877386 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e1b39fb-5dbvc\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:20.887912 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:20.887884 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56b92\" (UniqueName: \"kubernetes.io/projected/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-kube-api-access-56b92\") pod \"maas-api-revoke-sub-4e1b39fb-5dbvc\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:21.048284 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:21.048238 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:21.337551 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:21.337476 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc"] Nov 03 21:10:21.341652 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:10:21.341611 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0b174c_8a9e_4b0c_a514_97a34aa4ec4b.slice/crio-d0167a9a60c8acaa75a7c5c03b5a3b3d17d32ec7f5367376e39a1103bf696543 WatchSource:0}: Error finding container d0167a9a60c8acaa75a7c5c03b5a3b3d17d32ec7f5367376e39a1103bf696543: Status 404 returned error can't find the container with id d0167a9a60c8acaa75a7c5c03b5a3b3d17d32ec7f5367376e39a1103bf696543 Nov 03 21:10:21.471607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:21.471563 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" event={"ID":"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b","Type":"ContainerStarted","Data":"d0167a9a60c8acaa75a7c5c03b5a3b3d17d32ec7f5367376e39a1103bf696543"} Nov 03 21:10:22.505868 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:22.505835 4711 generic.go:356] "Generic (PLEG): container finished" podID="2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b" containerID="7c469759ac9060e180afd5e9ae735b34bd4662b335abfa633774df854ea8ab12" exitCode=0 Nov 03 21:10:22.506498 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:22.506470 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" event={"ID":"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b","Type":"ContainerDied","Data":"7c469759ac9060e180afd5e9ae735b34bd4662b335abfa633774df854ea8ab12"} Nov 03 21:10:23.685383 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.685361 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:23.853302 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.853226 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-kube-api-access-56b92\" (UniqueName: \"kubernetes.io/projected/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-kube-api-access-56b92\") pod \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " Nov 03 21:10:23.853302 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.853272 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-xks-ca\") pod \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " Nov 03 21:10:23.853569 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.853315 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-openshift-ca\") pod \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\" (UID: \"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b\") " Nov 03 21:10:23.853793 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.853766 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-openshift-ca" pod "2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b" (UID: "2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:23.855407 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.855388 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-kube-api-access-56b92" pod "2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b" (UID: "2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b"). InnerVolumeSpecName "kube-api-access-56b92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:23.855502 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.855437 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-xks-ca" pod "2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b" (UID: "2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:23.955239 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.955194 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56b92\" (UniqueName: \"kubernetes.io/projected/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-kube-api-access-56b92\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:23.955239 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.955235 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:23.955410 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:23.955250 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:24.522940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:24.522902 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" event={"ID":"2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b","Type":"ContainerDied","Data":"d0167a9a60c8acaa75a7c5c03b5a3b3d17d32ec7f5367376e39a1103bf696543"} Nov 03 21:10:24.522940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:24.522938 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc" Nov 03 21:10:24.523168 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:24.522961 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0167a9a60c8acaa75a7c5c03b5a3b3d17d32ec7f5367376e39a1103bf696543" Nov 03 21:10:30.900310 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:30.900267 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" podUID="9b2fef45-8c85-48ce-8575-ede14cb79ce4" containerName="main" containerID="cri-o://55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca" gracePeriod=45 Nov 03 21:10:31.161261 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.161198 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:10:31.225698 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.225668 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-model-cache\") pod \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " Nov 03 21:10:31.225881 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.225720 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2fef45-8c85-48ce-8575-ede14cb79ce4-kube-api-access-pcfbk\" (UniqueName: \"kubernetes.io/projected/9b2fef45-8c85-48ce-8575-ede14cb79ce4-kube-api-access-pcfbk\") pod \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " Nov 03 21:10:31.225881 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.225784 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tmp-dir\") pod \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " Nov 03 21:10:31.225881 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.225858 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-dshm\") pod \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " Nov 03 21:10:31.226060 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.225915 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tls-certs\") pod \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " Nov 03 21:10:31.226060 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.225945 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-home\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-home\") pod \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\" (UID: \"9b2fef45-8c85-48ce-8575-ede14cb79ce4\") " Nov 03 21:10:31.226060 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.225980 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-model-cache" pod "9b2fef45-8c85-48ce-8575-ede14cb79ce4" (UID: "9b2fef45-8c85-48ce-8575-ede14cb79ce4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:31.226202 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.226121 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tmp-dir" pod "9b2fef45-8c85-48ce-8575-ede14cb79ce4" (UID: "9b2fef45-8c85-48ce-8575-ede14cb79ce4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:31.226294 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.226266 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-home" pod "9b2fef45-8c85-48ce-8575-ede14cb79ce4" (UID: "9b2fef45-8c85-48ce-8575-ede14cb79ce4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:31.226514 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.226498 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:31.226606 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.226516 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:31.226606 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.226526 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:31.228131 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.228103 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tls-certs" pod "9b2fef45-8c85-48ce-8575-ede14cb79ce4" (UID: "9b2fef45-8c85-48ce-8575-ede14cb79ce4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:10:31.228131 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.228115 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-dshm" pod "9b2fef45-8c85-48ce-8575-ede14cb79ce4" (UID: "9b2fef45-8c85-48ce-8575-ede14cb79ce4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:31.228252 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.228137 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2fef45-8c85-48ce-8575-ede14cb79ce4-kube-api-access-pcfbk" pod "9b2fef45-8c85-48ce-8575-ede14cb79ce4" (UID: "9b2fef45-8c85-48ce-8575-ede14cb79ce4"). InnerVolumeSpecName "kube-api-access-pcfbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:31.327929 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.327891 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcfbk\" (UniqueName: \"kubernetes.io/projected/9b2fef45-8c85-48ce-8575-ede14cb79ce4-kube-api-access-pcfbk\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:31.327929 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.327922 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b2fef45-8c85-48ce-8575-ede14cb79ce4-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:31.327929 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.327932 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b2fef45-8c85-48ce-8575-ede14cb79ce4-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:31.560424 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.560388 4711 generic.go:356] "Generic (PLEG): container finished" podID="9b2fef45-8c85-48ce-8575-ede14cb79ce4" containerID="55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca" exitCode=0 Nov 03 21:10:31.560620 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.560463 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" Nov 03 21:10:31.560620 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.560477 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" event={"ID":"9b2fef45-8c85-48ce-8575-ede14cb79ce4","Type":"ContainerDied","Data":"55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca"} Nov 03 21:10:31.560620 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.560527 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz" event={"ID":"9b2fef45-8c85-48ce-8575-ede14cb79ce4","Type":"ContainerDied","Data":"74ee0514bff58ac8f93fe9a2f3d18f4184471d5bb287d8bcb980c7c1b60e58d9"} Nov 03 21:10:31.560620 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.560545 4711 scope.go:122] "RemoveContainer" containerID="55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca" Nov 03 21:10:31.569912 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.569897 4711 scope.go:122] "RemoveContainer" containerID="55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca" Nov 03 21:10:31.570171 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:10:31.570152 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca\": container with ID starting with 55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca not found: ID does not exist" containerID="55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca" Nov 03 21:10:31.570247 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.570178 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca"} err="failed to get container status \"55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca\": rpc error: code = NotFound desc = could not find container \"55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca\": container with ID starting with 55cf0d460f85a55f1512b4a789b91838883d0c6da81216c74b8e249332c3edca not found: ID does not exist" Nov 03 21:10:31.582660 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.582640 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz"] Nov 03 21:10:31.586410 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:31.586392 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-0704dddb-kserve-8b5c668dd-pmbhz"] Nov 03 21:10:32.469098 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.469066 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:32.678385 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.678342 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2fef45-8c85-48ce-8575-ede14cb79ce4" path="/var/lib/kubelet/pods/9b2fef45-8c85-48ce-8575-ede14cb79ce4/volumes" Nov 03 21:10:32.831525 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.831494 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g"] Nov 03 21:10:32.831964 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.831950 4711 cpu_manager.go:414] "containerMap: removing container" podUID="2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b" containerName="revoke-keys" Nov 03 21:10:32.832007 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.831965 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b" containerName="revoke-keys" Nov 03 21:10:32.832007 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.831993 4711 cpu_manager.go:414] "containerMap: removing container" podUID="9b2fef45-8c85-48ce-8575-ede14cb79ce4" containerName="main" Nov 03 21:10:32.832007 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.832001 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2fef45-8c85-48ce-8575-ede14cb79ce4" containerName="main" Nov 03 21:10:32.832106 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.832082 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2fef45-8c85-48ce-8575-ede14cb79ce4" containerName="main" Nov 03 21:10:32.832106 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.832095 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b" containerName="revoke-keys" Nov 03 21:10:32.836659 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.836638 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:32.843863 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.843738 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-debeb1-openshift-default-dockercfg-62sbp\"" Nov 03 21:10:32.851151 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.851125 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g"] Nov 03 21:10:32.917774 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.917729 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:32.944373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.944345 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-podinfo\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:32.944528 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.944384 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istiod-ca-cert\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:32.944528 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.944486 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-envoy\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:32.944625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.944527 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-token\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:32.944625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.944577 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-certs\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:32.944625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.944619 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-credential-socket\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:32.944727 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.944640 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-socket\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:32.944787 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.944770 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-data\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:32.944824 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:32.944808 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8v24\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-kube-api-access-s8v24\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.045817 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.045783 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-podinfo\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046031 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.045826 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istiod-ca-cert\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046116 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046097 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-envoy\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046116 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046114 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-token\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046236 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046147 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-certs\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046236 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046217 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-credential-socket\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046345 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046250 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-socket\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046345 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046307 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-data\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046480 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046352 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8v24\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-kube-api-access-s8v24\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046833 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046589 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-certs\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046833 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046787 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-credential-socket\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.046833 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046820 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istiod-ca-cert\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.047003 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.046979 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-socket\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.047173 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.047142 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-data\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.049672 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.049606 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-envoy\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.049809 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.049670 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-podinfo\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.051265 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.050849 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:33.058882 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.058856 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-token\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.059189 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.059168 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8v24\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-kube-api-access-s8v24\") pod \"e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.150984 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.150911 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:33.435403 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.435373 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g"] Nov 03 21:10:33.437009 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:10:33.436982 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50ea865_76cc_45f8_aa8d_1dd4ad300707.slice/crio-0baf6f09b3a1fdff3b0bb895a80b58f9167e37faf2dc719272afef479aff8f3f WatchSource:0}: Error finding container 0baf6f09b3a1fdff3b0bb895a80b58f9167e37faf2dc719272afef479aff8f3f: Status 404 returned error can't find the container with id 0baf6f09b3a1fdff3b0bb895a80b58f9167e37faf2dc719272afef479aff8f3f Nov 03 21:10:33.439411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.439377 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:33.439531 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.439509 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:33.439593 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.439577 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:33.576137 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.576104 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" event={"ID":"b50ea865-76cc-45f8-aa8d-1dd4ad300707","Type":"ContainerStarted","Data":"51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788"} Nov 03 21:10:33.576539 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.576190 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" event={"ID":"b50ea865-76cc-45f8-aa8d-1dd4ad300707","Type":"ContainerStarted","Data":"0baf6f09b3a1fdff3b0bb895a80b58f9167e37faf2dc719272afef479aff8f3f"} Nov 03 21:10:33.601134 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:33.601082 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" podStartSLOduration=1.601067555 podStartE2EDuration="1.601067555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:33.598770856 +0000 UTC m=+4137.048876902" watchObservedRunningTime="2026-11-03 21:10:33.601067555 +0000 UTC m=+4137.051173658" Nov 03 21:10:34.152034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:34.152000 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:34.157202 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:34.157174 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:34.582071 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:34.582042 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:34.583121 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:34.583101 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:34.685843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:34.685806 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:35.870971 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:35.870935 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:36.008338 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:36.008299 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:37.042437 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.042401 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq"] Nov 03 21:10:37.046592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.046568 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.057921 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.057178 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:10:37.057921 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.057246 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:10:37.057921 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.057847 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqpwn\"" Nov 03 21:10:37.057921 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.057885 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-debeb1-serving-cert\"" Nov 03 21:10:37.057921 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.057248 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:10:37.060397 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.060378 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq"] Nov 03 21:10:37.084292 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.084185 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:37.091439 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.091407 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.091604 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.091532 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.091604 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.091571 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6efc79b0-e24f-4a4c-bf38-f291e291abb2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.091604 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.091589 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqv7\" (UniqueName: \"kubernetes.io/projected/6efc79b0-e24f-4a4c-bf38-f291e291abb2-kube-api-access-8rqv7\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.128717 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.128689 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5"] Nov 03 21:10:37.132529 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.132510 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:37.143821 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.143801 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5"] Nov 03 21:10:37.155942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.155919 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-debeb1\"" Nov 03 21:10:37.156039 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.155919 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-de9dd262-dockercfg-rsdjv\"" Nov 03 21:10:37.166532 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.166511 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4"] Nov 03 21:10:37.170234 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.170215 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:37.182364 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.182340 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4"] Nov 03 21:10:37.192606 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.192575 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5\" (UID: \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:37.192740 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.192698 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.192740 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.192730 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnww\" (UniqueName: \"kubernetes.io/projected/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-kube-api-access-zpnww\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5\" (UID: \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:37.192903 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.192763 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.192903 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.192787 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6efc79b0-e24f-4a4c-bf38-f291e291abb2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.192903 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.192803 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqv7\" (UniqueName: \"kubernetes.io/projected/6efc79b0-e24f-4a4c-bf38-f291e291abb2-kube-api-access-8rqv7\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.193619 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.193595 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6efc79b0-e24f-4a4c-bf38-f291e291abb2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.195221 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.195195 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.195331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.195228 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.204856 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.204832 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqv7\" (UniqueName: \"kubernetes.io/projected/6efc79b0-e24f-4a4c-bf38-f291e291abb2-kube-api-access-8rqv7\") pod \"maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.220303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.220273 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:37.293822 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.293730 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d3409-07ce-4d0f-a0d6-0dcb62551922-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4\" (UID: \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:37.293822 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.293778 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5\" (UID: \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:37.293822 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.293814 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbsp2\" (UniqueName: \"kubernetes.io/projected/f28d3409-07ce-4d0f-a0d6-0dcb62551922-kube-api-access-tbsp2\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4\" (UID: \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:37.294138 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.293965 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnww\" (UniqueName: \"kubernetes.io/projected/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-kube-api-access-zpnww\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5\" (UID: \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:37.294599 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.294574 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5\" (UID: \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:37.306357 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.306330 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnww\" (UniqueName: \"kubernetes.io/projected/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-kube-api-access-zpnww\") pod \"payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5\" (UID: \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:37.340843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.340772 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:37.364994 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.364972 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:37.395578 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.395549 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d3409-07ce-4d0f-a0d6-0dcb62551922-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4\" (UID: \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:37.395712 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.395603 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbsp2\" (UniqueName: \"kubernetes.io/projected/f28d3409-07ce-4d0f-a0d6-0dcb62551922-kube-api-access-tbsp2\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4\" (UID: \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:37.396226 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.396207 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d3409-07ce-4d0f-a0d6-0dcb62551922-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4\" (UID: \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:37.407973 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.407940 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbsp2\" (UniqueName: \"kubernetes.io/projected/f28d3409-07ce-4d0f-a0d6-0dcb62551922-kube-api-access-tbsp2\") pod \"payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4\" (UID: \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:37.446016 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.445988 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:37.487639 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.487071 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:37.642108 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.642085 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq"] Nov 03 21:10:37.644235 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:10:37.644202 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efc79b0_e24f_4a4c_bf38_f291e291abb2.slice/crio-d41b836dc68b4470bac473ea66eff2524c35192ff9d7864f63c578896bba7e15 WatchSource:0}: Error finding container d41b836dc68b4470bac473ea66eff2524c35192ff9d7864f63c578896bba7e15: Status 404 returned error can't find the container with id d41b836dc68b4470bac473ea66eff2524c35192ff9d7864f63c578896bba7e15 Nov 03 21:10:37.645659 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.645642 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:10:37.726269 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.726238 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5"] Nov 03 21:10:37.728293 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:10:37.728265 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd468fb4_63e9_4e0f_be2d_73de9b58f1ab.slice/crio-b2b5f11cd69245b9d70b87e6157a3467a66169a9a5e06ea04113d7210ab2d427 WatchSource:0}: Error finding container b2b5f11cd69245b9d70b87e6157a3467a66169a9a5e06ea04113d7210ab2d427: Status 404 returned error can't find the container with id b2b5f11cd69245b9d70b87e6157a3467a66169a9a5e06ea04113d7210ab2d427 Nov 03 21:10:37.790686 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:37.790659 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4"] Nov 03 21:10:37.792685 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:10:37.792660 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28d3409_07ce_4d0f_a0d6_0dcb62551922.slice/crio-83c3082eb8611bcc477b25d38d1fdfb62f115e293b16436483e28f15ffab4c4d WatchSource:0}: Error finding container 83c3082eb8611bcc477b25d38d1fdfb62f115e293b16436483e28f15ffab4c4d: Status 404 returned error can't find the container with id 83c3082eb8611bcc477b25d38d1fdfb62f115e293b16436483e28f15ffab4c4d Nov 03 21:10:38.009181 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.009144 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:38.606701 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.606665 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" event={"ID":"f28d3409-07ce-4d0f-a0d6-0dcb62551922","Type":"ContainerStarted","Data":"5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849"} Nov 03 21:10:38.607137 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.606730 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:38.607137 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.606755 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" event={"ID":"f28d3409-07ce-4d0f-a0d6-0dcb62551922","Type":"ContainerStarted","Data":"83c3082eb8611bcc477b25d38d1fdfb62f115e293b16436483e28f15ffab4c4d"} Nov 03 21:10:38.608273 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.608242 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" event={"ID":"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab","Type":"ContainerStarted","Data":"210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479"} Nov 03 21:10:38.608421 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.608293 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" event={"ID":"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab","Type":"ContainerStarted","Data":"b2b5f11cd69245b9d70b87e6157a3467a66169a9a5e06ea04113d7210ab2d427"} Nov 03 21:10:38.608421 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.608352 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:38.609827 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.609806 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" event={"ID":"6efc79b0-e24f-4a4c-bf38-f291e291abb2","Type":"ContainerStarted","Data":"b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46"} Nov 03 21:10:38.609951 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.609839 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" event={"ID":"6efc79b0-e24f-4a4c-bf38-f291e291abb2","Type":"ContainerStarted","Data":"d41b836dc68b4470bac473ea66eff2524c35192ff9d7864f63c578896bba7e15"} Nov 03 21:10:38.609951 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.609896 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:38.627013 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.626962 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" podStartSLOduration=1.626947206 podStartE2EDuration="1.626947206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:38.624327058 +0000 UTC m=+4142.074433103" watchObservedRunningTime="2026-11-03 21:10:38.626947206 +0000 UTC m=+4142.077053254" Nov 03 21:10:38.646365 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.646315 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" podStartSLOduration=1.170109456 podStartE2EDuration="1.64629958s" totalImagesPullingTime="476.190124ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:38.644159087 +0000 UTC m=+4142.094265134" watchObservedRunningTime="2026-11-03 21:10:38.64629958 +0000 UTC m=+4142.096405626" Nov 03 21:10:38.664463 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.664404 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" podStartSLOduration=1.66439067 podStartE2EDuration="1.66439067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:38.663738827 +0000 UTC m=+4142.113844887" watchObservedRunningTime="2026-11-03 21:10:38.66439067 +0000 UTC m=+4142.114496716" Nov 03 21:10:38.817870 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:38.817830 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:39.594978 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:39.594945 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:40.385626 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:40.385595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:41.095670 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:41.095630 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:41.914437 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:41.914403 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:42.604162 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:42.604124 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:43.310506 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:43.310465 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:44.010507 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:44.010465 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:44.621278 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:44.621250 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:10:44.714665 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:44.714631 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:45.498230 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:45.498187 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:46.209015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:46.208984 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:47.014837 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:47.014800 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:47.534176 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:10:47.534126 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:10:47.710802 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:47.710768 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:48.405290 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:48.405224 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:49.109067 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.109037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:49.583670 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.583637 4711 patch_prober.go:29] interesting pod/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 21:10:49.583850 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.583688 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 21:10:49.618877 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.618850 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:10:49.619037 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.618925 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:10:49.676356 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.676322 4711 generic.go:356] "Generic (PLEG): container finished" podID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerID="51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788" exitCode=0 Nov 03 21:10:49.676575 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.676364 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" event={"ID":"b50ea865-76cc-45f8-aa8d-1dd4ad300707","Type":"ContainerDied","Data":"51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788"} Nov 03 21:10:49.676882 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.676860 4711 scope.go:122] "RemoveContainer" containerID="51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788" Nov 03 21:10:49.678149 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.678120 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:49.678248 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.678184 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:49.678248 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.678218 4711 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 21:10:49.814807 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:49.814774 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:50.591711 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:50.591673 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:50.699626 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:50.699587 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" event={"ID":"b50ea865-76cc-45f8-aa8d-1dd4ad300707","Type":"ContainerStarted","Data":"0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b"} Nov 03 21:10:50.891095 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:50.890995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:51.151283 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:51.151204 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:51.151440 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:51.151377 4711 patch_prober.go:29] interesting pod/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 21:10:51.151440 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:51.151427 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 21:10:51.304144 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:51.304114 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:51.511939 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:51.511904 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:51.644389 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:51.644355 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:51.788479 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:51.788358 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:52.124734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.124649 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:52.152398 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.152363 4711 patch_prober.go:29] interesting pod/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 21:10:52.152594 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.152437 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 21:10:52.499389 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.499301 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n"] Nov 03 21:10:52.503498 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.503473 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n"] Nov 03 21:10:52.503641 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.503561 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.512998 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.512973 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-debeb1\"/\"simulator-single-main-debeb1-kserve-self-signed-certs\"" Nov 03 21:10:52.514094 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.514051 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-debeb1\"/\"default-dockercfg-xmqx9\"" Nov 03 21:10:52.514094 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.514071 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-debeb1\"/\"openshift-service-ca.crt\"" Nov 03 21:10:52.514253 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.514126 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-debeb1\"/\"kube-root-ca.crt\"" Nov 03 21:10:52.591362 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.591317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:52.601897 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.601867 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4r9\" (UniqueName: \"kubernetes.io/projected/f0c02244-2313-495c-a0e9-52d7a1b9636b-kube-api-access-fk4r9\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.602046 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.601903 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-model-cache\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.602046 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.601928 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-tmp-dir\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.602140 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.602055 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-home\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.602140 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.602097 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c02244-2313-495c-a0e9-52d7a1b9636b-tls-certs\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.602255 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.602157 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-dshm\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.702899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.702866 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c02244-2313-495c-a0e9-52d7a1b9636b-tls-certs\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.703327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.703107 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-dshm\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.703327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.703180 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4r9\" (UniqueName: \"kubernetes.io/projected/f0c02244-2313-495c-a0e9-52d7a1b9636b-kube-api-access-fk4r9\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.703327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.703205 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-model-cache\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.703327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.703221 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-tmp-dir\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.703327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.703319 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-home\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.703625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.703571 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-tmp-dir\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.703832 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.703810 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-home\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.703922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.703851 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-model-cache\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.705387 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.705344 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-dshm\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.705650 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.705630 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c02244-2313-495c-a0e9-52d7a1b9636b-tls-certs\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.715761 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.715737 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4r9\" (UniqueName: \"kubernetes.io/projected/f0c02244-2313-495c-a0e9-52d7a1b9636b-kube-api-access-fk4r9\") pod \"simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.818427 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.818390 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:52.909490 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:52.909341 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:53.120922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:53.120893 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n"] Nov 03 21:10:53.122791 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:10:53.122761 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0c02244_2313_495c_a0e9_52d7a1b9636b.slice/crio-bcdb693767e870dd3d9ca01d9e48319daa24f76252c498ed8cb378bdf4f5f2f7 WatchSource:0}: Error finding container bcdb693767e870dd3d9ca01d9e48319daa24f76252c498ed8cb378bdf4f5f2f7: Status 404 returned error can't find the container with id bcdb693767e870dd3d9ca01d9e48319daa24f76252c498ed8cb378bdf4f5f2f7 Nov 03 21:10:53.151877 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:53.151855 4711 patch_prober.go:29] interesting pod/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 21:10:53.151970 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:53.151905 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 21:10:53.621013 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:53.620973 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:53.717079 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:53.717041 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" event={"ID":"f0c02244-2313-495c-a0e9-52d7a1b9636b","Type":"ContainerStarted","Data":"e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82"} Nov 03 21:10:53.717479 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:53.717086 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" event={"ID":"f0c02244-2313-495c-a0e9-52d7a1b9636b","Type":"ContainerStarted","Data":"bcdb693767e870dd3d9ca01d9e48319daa24f76252c498ed8cb378bdf4f5f2f7"} Nov 03 21:10:53.819469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:53.819425 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:10:54.151703 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:54.151670 4711 patch_prober.go:29] interesting pod/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 21:10:54.151885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:54.151721 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 21:10:54.402640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:54.402555 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:55.152297 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:55.152257 4711 patch_prober.go:29] interesting pod/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 21:10:55.152833 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:55.152322 4711 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 21:10:55.228733 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:55.228683 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:55.915067 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:55.915018 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:56.156201 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:56.156171 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:56.156679 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:56.156666 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:56.157474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:56.157431 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:10:56.183699 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:56.183613 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" podStartSLOduration=4.183598997 podStartE2EDuration="4.183598997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:53.790704608 +0000 UTC m=+4157.240810657" watchObservedRunningTime="2026-11-03 21:10:56.183598997 +0000 UTC m=+4159.633705077" Nov 03 21:10:56.714851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:56.714815 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:57.520085 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:57.520039 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:58.234560 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:58.234525 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:59.110403 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:59.110352 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:10:59.810719 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:10:59.810689 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:00.284356 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:00.284319 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:00.574735 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:00.574644 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:01.207173 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:01.207137 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:01.557183 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:01.557149 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:01.902717 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:01.902615 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:02.618086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:02.618048 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:02.818860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:02.818822 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:11:02.831522 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:02.831490 4711 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:11:03.313475 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:03.313424 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:03.783327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:03.783296 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:11:04.007631 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:04.007591 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:04.818823 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:04.818783 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:05.519196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:05.519138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:06.213950 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:06.213910 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:06.919305 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:06.919268 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:08.374332 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:08.374296 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:09.224012 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:09.223974 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:09.918479 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:09.918421 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:10.824389 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:10.824355 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:11.599729 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:11.599694 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:12.327535 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:12.327436 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:13.108095 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:13.108061 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:13.811064 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:13.811007 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:14.507677 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:14.507639 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:15.220193 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:15.220156 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:15.911252 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:15.911218 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:16.604473 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:16.604422 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:17.450377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:17.450343 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:17.531808 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:11:17.531768 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:11:18.111423 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:18.111386 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:18.813526 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:18.813492 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:19.509849 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:19.509813 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:20.213684 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:20.213647 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:20.904243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:20.904186 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:21.603855 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:21.603813 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:22.302663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:22.302619 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:23.012524 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:23.012485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:23.706487 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:23.706430 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:24.495699 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:24.495662 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:25.298711 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:25.298677 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:26.003354 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:26.003310 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:26.695996 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:26.695966 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:27.395474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:27.394877 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:28.114731 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:28.114690 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:28.808283 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:28.808247 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:29.686839 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:29.686802 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:29.818293 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:29.818252 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:30.387069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:30.387033 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:31.094574 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:31.094532 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:31.897860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:31.897823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:32.598663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:32.598629 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:33.293868 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:33.293823 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:33.830017 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:33.829980 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:34.023331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:34.023292 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:34.314654 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:34.314624 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:34.711532 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:34.711429 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:35.402187 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:35.402155 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:36.104490 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:36.104436 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:36.794515 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:36.794483 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:37.499258 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:37.499227 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:37.922432 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:37.922408 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:11:37.925313 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:37.925294 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:11:37.968318 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:37.968292 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:11:37.971108 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:37.971084 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:11:38.267706 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:38.267662 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:38.757624 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:38.757594 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n"] Nov 03 21:11:38.862426 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:38.862382 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:39.103481 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:39.103339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:39.806069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:39.806032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:39.942982 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:39.942940 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4"] Nov 03 21:11:39.957391 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:39.957358 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5"] Nov 03 21:11:40.053084 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.053052 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:40.189595 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.189498 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:40.395429 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.395396 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq"] Nov 03 21:11:40.395628 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.395609 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" podUID="6efc79b0-e24f-4a4c-bf38-f291e291abb2" containerName="maas-api" containerID="cri-o://b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46" gracePeriod=30 Nov 03 21:11:40.483233 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.483146 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:40.843951 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.843924 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:11:40.874227 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.874197 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:40.920731 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.920708 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-maas-api-tls\") pod \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " Nov 03 21:11:40.920899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.920736 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-metrics-certs\") pod \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " Nov 03 21:11:40.920899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.920834 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6efc79b0-e24f-4a4c-bf38-f291e291abb2-kube-api-access-8rqv7\" (UniqueName: \"kubernetes.io/projected/6efc79b0-e24f-4a4c-bf38-f291e291abb2-kube-api-access-8rqv7\") pod \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " Nov 03 21:11:40.920899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.920895 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6efc79b0-e24f-4a4c-bf38-f291e291abb2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6efc79b0-e24f-4a4c-bf38-f291e291abb2-openshift-service-ca\") pod \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\" (UID: \"6efc79b0-e24f-4a4c-bf38-f291e291abb2\") " Nov 03 21:11:40.921325 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.921296 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6efc79b0-e24f-4a4c-bf38-f291e291abb2-openshift-service-ca" pod "6efc79b0-e24f-4a4c-bf38-f291e291abb2" (UID: "6efc79b0-e24f-4a4c-bf38-f291e291abb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:40.922910 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.922883 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-maas-api-tls" pod "6efc79b0-e24f-4a4c-bf38-f291e291abb2" (UID: "6efc79b0-e24f-4a4c-bf38-f291e291abb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:40.922910 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.922894 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efc79b0-e24f-4a4c-bf38-f291e291abb2-kube-api-access-8rqv7" pod "6efc79b0-e24f-4a4c-bf38-f291e291abb2" (UID: "6efc79b0-e24f-4a4c-bf38-f291e291abb2"). InnerVolumeSpecName "kube-api-access-8rqv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:40.923086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.922948 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-metrics-certs" pod "6efc79b0-e24f-4a4c-bf38-f291e291abb2" (UID: "6efc79b0-e24f-4a4c-bf38-f291e291abb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:40.977373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.977339 4711 generic.go:356] "Generic (PLEG): container finished" podID="6efc79b0-e24f-4a4c-bf38-f291e291abb2" containerID="b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46" exitCode=0 Nov 03 21:11:40.977572 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.977382 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" Nov 03 21:11:40.977572 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.977418 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" event={"ID":"6efc79b0-e24f-4a4c-bf38-f291e291abb2","Type":"ContainerDied","Data":"b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46"} Nov 03 21:11:40.977572 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.977487 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq" event={"ID":"6efc79b0-e24f-4a4c-bf38-f291e291abb2","Type":"ContainerDied","Data":"d41b836dc68b4470bac473ea66eff2524c35192ff9d7864f63c578896bba7e15"} Nov 03 21:11:40.977572 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.977510 4711 scope.go:122] "RemoveContainer" containerID="b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46" Nov 03 21:11:40.987064 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.987047 4711 scope.go:122] "RemoveContainer" containerID="b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46" Nov 03 21:11:40.987331 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:11:40.987311 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46\": container with ID starting with b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46 not found: ID does not exist" containerID="b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46" Nov 03 21:11:40.987378 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:40.987341 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46"} err="failed to get container status \"b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46\": rpc error: code = NotFound desc = could not find container \"b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46\": container with ID starting with b6a434dbc3364a5407b8fd82eaaf81244b31d7368af875474d20c59b0e797e46 not found: ID does not exist" Nov 03 21:11:41.000496 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:41.000473 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq"] Nov 03 21:11:41.004689 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:41.004655 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-debeb1-6b9f4445-p59pq"] Nov 03 21:11:41.022461 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:41.022398 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6efc79b0-e24f-4a4c-bf38-f291e291abb2-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:41.022461 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:41.022421 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:41.022461 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:41.022430 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6efc79b0-e24f-4a4c-bf38-f291e291abb2-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:41.022461 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:41.022439 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rqv7\" (UniqueName: \"kubernetes.io/projected/6efc79b0-e24f-4a4c-bf38-f291e291abb2-kube-api-access-8rqv7\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:42.679086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:42.679052 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6efc79b0-e24f-4a4c-bf38-f291e291abb2" path="/var/lib/kubelet/pods/6efc79b0-e24f-4a4c-bf38-f291e291abb2/volumes" Nov 03 21:11:44.944219 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:44.944170 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" podUID="f28d3409-07ce-4d0f-a0d6-0dcb62551922" containerName="payload-processing" containerID="cri-o://5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849" gracePeriod=25 Nov 03 21:11:44.958356 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:44.958305 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" podUID="bd468fb4-63e9-4e0f-be2d-73de9b58f1ab" containerName="payload-pre-processing" containerID="cri-o://210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479" gracePeriod=25 Nov 03 21:11:45.238258 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.238231 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:11:45.240279 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.240259 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:11:45.361935 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.361904 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-plugins-config-volume\") pod \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\" (UID: \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\") " Nov 03 21:11:45.362113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.361988 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28d3409-07ce-4d0f-a0d6-0dcb62551922-kube-api-access-tbsp2\" (UniqueName: \"kubernetes.io/projected/f28d3409-07ce-4d0f-a0d6-0dcb62551922-kube-api-access-tbsp2\") pod \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\" (UID: \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\") " Nov 03 21:11:45.362113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.362083 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-kube-api-access-zpnww\" (UniqueName: \"kubernetes.io/projected/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-kube-api-access-zpnww\") pod \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\" (UID: \"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab\") " Nov 03 21:11:45.362198 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.362187 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28d3409-07ce-4d0f-a0d6-0dcb62551922-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d3409-07ce-4d0f-a0d6-0dcb62551922-plugins-config-volume\") pod \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\" (UID: \"f28d3409-07ce-4d0f-a0d6-0dcb62551922\") " Nov 03 21:11:45.362373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.362347 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-plugins-config-volume" pod "bd468fb4-63e9-4e0f-be2d-73de9b58f1ab" (UID: "bd468fb4-63e9-4e0f-be2d-73de9b58f1ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:45.362606 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.362581 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28d3409-07ce-4d0f-a0d6-0dcb62551922-plugins-config-volume" pod "f28d3409-07ce-4d0f-a0d6-0dcb62551922" (UID: "f28d3409-07ce-4d0f-a0d6-0dcb62551922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:45.362725 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.362699 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:45.362725 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.362712 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28d3409-07ce-4d0f-a0d6-0dcb62551922-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:45.364172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.364149 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-kube-api-access-zpnww" pod "bd468fb4-63e9-4e0f-be2d-73de9b58f1ab" (UID: "bd468fb4-63e9-4e0f-be2d-73de9b58f1ab"). InnerVolumeSpecName "kube-api-access-zpnww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:45.364269 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.364177 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28d3409-07ce-4d0f-a0d6-0dcb62551922-kube-api-access-tbsp2" pod "f28d3409-07ce-4d0f-a0d6-0dcb62551922" (UID: "f28d3409-07ce-4d0f-a0d6-0dcb62551922"). InnerVolumeSpecName "kube-api-access-tbsp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:45.463683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.463602 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbsp2\" (UniqueName: \"kubernetes.io/projected/f28d3409-07ce-4d0f-a0d6-0dcb62551922-kube-api-access-tbsp2\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:45.463683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:45.463633 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpnww\" (UniqueName: \"kubernetes.io/projected/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab-kube-api-access-zpnww\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:46.005406 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.005375 4711 generic.go:356] "Generic (PLEG): container finished" podID="f28d3409-07ce-4d0f-a0d6-0dcb62551922" containerID="5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849" exitCode=0 Nov 03 21:11:46.005833 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.005417 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" Nov 03 21:11:46.005833 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.005471 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" event={"ID":"f28d3409-07ce-4d0f-a0d6-0dcb62551922","Type":"ContainerDied","Data":"5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849"} Nov 03 21:11:46.005833 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.005540 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4" event={"ID":"f28d3409-07ce-4d0f-a0d6-0dcb62551922","Type":"ContainerDied","Data":"83c3082eb8611bcc477b25d38d1fdfb62f115e293b16436483e28f15ffab4c4d"} Nov 03 21:11:46.005833 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.005574 4711 scope.go:122] "RemoveContainer" containerID="5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849" Nov 03 21:11:46.006942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.006882 4711 generic.go:356] "Generic (PLEG): container finished" podID="bd468fb4-63e9-4e0f-be2d-73de9b58f1ab" containerID="210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479" exitCode=0 Nov 03 21:11:46.006942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.006926 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" Nov 03 21:11:46.007073 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.006925 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" event={"ID":"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab","Type":"ContainerDied","Data":"210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479"} Nov 03 21:11:46.007073 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.007051 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5" event={"ID":"bd468fb4-63e9-4e0f-be2d-73de9b58f1ab","Type":"ContainerDied","Data":"b2b5f11cd69245b9d70b87e6157a3467a66169a9a5e06ea04113d7210ab2d427"} Nov 03 21:11:46.016236 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.016216 4711 scope.go:122] "RemoveContainer" containerID="5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849" Nov 03 21:11:46.016540 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:11:46.016516 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849\": container with ID starting with 5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849 not found: ID does not exist" containerID="5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849" Nov 03 21:11:46.016640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.016545 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849"} err="failed to get container status \"5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849\": rpc error: code = NotFound desc = could not find container \"5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849\": container with ID starting with 5b8b3362d41d22de67d350b97914f9622e7e149b2145e2c6fd64c0d949c0c849 not found: ID does not exist" Nov 03 21:11:46.016640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.016566 4711 scope.go:122] "RemoveContainer" containerID="210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479" Nov 03 21:11:46.024664 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.024648 4711 scope.go:122] "RemoveContainer" containerID="210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479" Nov 03 21:11:46.024916 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:11:46.024899 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479\": container with ID starting with 210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479 not found: ID does not exist" containerID="210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479" Nov 03 21:11:46.024985 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.024925 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479"} err="failed to get container status \"210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479\": rpc error: code = NotFound desc = could not find container \"210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479\": container with ID starting with 210d7cab0ffda54f1ece6186dbfdf60a7e278ca3e95b355c1d6eaa2c2a86e479 not found: ID does not exist" Nov 03 21:11:46.030640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.030617 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4"] Nov 03 21:11:46.034816 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.034795 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-debeb1-85975f44fl4wb4"] Nov 03 21:11:46.045828 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.045802 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5"] Nov 03 21:11:46.047945 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.047925 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-debeb1-74694f98m5"] Nov 03 21:11:46.596912 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.596881 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:46.678642 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.678612 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd468fb4-63e9-4e0f-be2d-73de9b58f1ab" path="/var/lib/kubelet/pods/bd468fb4-63e9-4e0f-be2d-73de9b58f1ab/volumes" Nov 03 21:11:46.678977 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.678964 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28d3409-07ce-4d0f-a0d6-0dcb62551922" path="/var/lib/kubelet/pods/f28d3409-07ce-4d0f-a0d6-0dcb62551922/volumes" Nov 03 21:11:46.730771 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:46.730738 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:11:47.531484 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:11:47.531427 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:11:53.777455 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:53.777398 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" podUID="f0c02244-2313-495c-a0e9-52d7a1b9636b" containerName="main" containerID="cri-o://e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82" gracePeriod=45 Nov 03 21:11:53.781683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:53.781653 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" podUID="f0c02244-2313-495c-a0e9-52d7a1b9636b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.195:8000/ready\": EOF" Nov 03 21:11:54.032262 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.032200 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:11:54.054414 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.054387 4711 generic.go:356] "Generic (PLEG): container finished" podID="f0c02244-2313-495c-a0e9-52d7a1b9636b" containerID="e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82" exitCode=0 Nov 03 21:11:54.054589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.054438 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" Nov 03 21:11:54.054589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.054440 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" event={"ID":"f0c02244-2313-495c-a0e9-52d7a1b9636b","Type":"ContainerDied","Data":"e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82"} Nov 03 21:11:54.054589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.054568 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n" event={"ID":"f0c02244-2313-495c-a0e9-52d7a1b9636b","Type":"ContainerDied","Data":"bcdb693767e870dd3d9ca01d9e48319daa24f76252c498ed8cb378bdf4f5f2f7"} Nov 03 21:11:54.054589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.054584 4711 scope.go:122] "RemoveContainer" containerID="e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82" Nov 03 21:11:54.065692 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.065672 4711 scope.go:122] "RemoveContainer" containerID="e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82" Nov 03 21:11:54.066027 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:11:54.065998 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82\": container with ID starting with e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82 not found: ID does not exist" containerID="e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82" Nov 03 21:11:54.066123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.066041 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82"} err="failed to get container status \"e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82\": rpc error: code = NotFound desc = could not find container \"e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82\": container with ID starting with e082a8bd6ca1f0c68f201f8ee265a84e56db6adbec56c010f113f08e82d05a82 not found: ID does not exist" Nov 03 21:11:54.154655 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.154623 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-tmp-dir\") pod \"f0c02244-2313-495c-a0e9-52d7a1b9636b\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " Nov 03 21:11:54.154848 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.154699 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-model-cache\") pod \"f0c02244-2313-495c-a0e9-52d7a1b9636b\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " Nov 03 21:11:54.154848 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.154798 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c02244-2313-495c-a0e9-52d7a1b9636b-kube-api-access-fk4r9\" (UniqueName: \"kubernetes.io/projected/f0c02244-2313-495c-a0e9-52d7a1b9636b-kube-api-access-fk4r9\") pod \"f0c02244-2313-495c-a0e9-52d7a1b9636b\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " Nov 03 21:11:54.154848 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.154842 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-home\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-home\") pod \"f0c02244-2313-495c-a0e9-52d7a1b9636b\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " Nov 03 21:11:54.155025 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.154887 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-tmp-dir" pod "f0c02244-2313-495c-a0e9-52d7a1b9636b" (UID: "f0c02244-2313-495c-a0e9-52d7a1b9636b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:54.155025 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.154922 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0c02244-2313-495c-a0e9-52d7a1b9636b-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c02244-2313-495c-a0e9-52d7a1b9636b-tls-certs\") pod \"f0c02244-2313-495c-a0e9-52d7a1b9636b\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " Nov 03 21:11:54.155025 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.154949 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-model-cache" pod "f0c02244-2313-495c-a0e9-52d7a1b9636b" (UID: "f0c02244-2313-495c-a0e9-52d7a1b9636b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:54.155025 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.154972 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-dshm\") pod \"f0c02244-2313-495c-a0e9-52d7a1b9636b\" (UID: \"f0c02244-2313-495c-a0e9-52d7a1b9636b\") " Nov 03 21:11:54.155234 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.155068 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-home" pod "f0c02244-2313-495c-a0e9-52d7a1b9636b" (UID: "f0c02244-2313-495c-a0e9-52d7a1b9636b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:54.155344 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.155326 4711 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-tmp-dir\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:54.155411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.155349 4711 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-model-cache\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:54.155411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.155363 4711 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-home\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:54.156981 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.156946 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c02244-2313-495c-a0e9-52d7a1b9636b-tls-certs" pod "f0c02244-2313-495c-a0e9-52d7a1b9636b" (UID: "f0c02244-2313-495c-a0e9-52d7a1b9636b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:54.156981 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.156961 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-dshm" pod "f0c02244-2313-495c-a0e9-52d7a1b9636b" (UID: "f0c02244-2313-495c-a0e9-52d7a1b9636b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:54.157133 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.156994 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c02244-2313-495c-a0e9-52d7a1b9636b-kube-api-access-fk4r9" pod "f0c02244-2313-495c-a0e9-52d7a1b9636b" (UID: "f0c02244-2313-495c-a0e9-52d7a1b9636b"). InnerVolumeSpecName "kube-api-access-fk4r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:54.255917 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.255886 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk4r9\" (UniqueName: \"kubernetes.io/projected/f0c02244-2313-495c-a0e9-52d7a1b9636b-kube-api-access-fk4r9\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:54.255917 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.255919 4711 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c02244-2313-495c-a0e9-52d7a1b9636b-tls-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:54.256176 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.255936 4711 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c02244-2313-495c-a0e9-52d7a1b9636b-dshm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:54.379077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.379050 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n"] Nov 03 21:11:54.385042 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.385013 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-debeb1/simulator-single-main-debeb1-kserve-7fdddf4f97-7wh4n"] Nov 03 21:11:54.679057 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:11:54.678979 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c02244-2313-495c-a0e9-52d7a1b9636b" path="/var/lib/kubelet/pods/f0c02244-2313-495c-a0e9-52d7a1b9636b/volumes" Nov 03 21:12:00.340454 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:00.340408 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:01.602556 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:01.602518 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:10.660247 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:10.660209 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:10.799064 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:10.799031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:11.865666 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:11.865630 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:12.018963 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:12.018928 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:12.154383 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:12.154306 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:12.712883 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:12.712831 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:13.413114 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:13.413081 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:14.026126 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:14.026091 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb"] Nov 03 21:12:14.028129 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:14.028109 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-x4jqb"] Nov 03 21:12:14.101970 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:14.101941 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:14.678907 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:14.678873 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4" path="/var/lib/kubelet/pods/2fa4d95b-ee33-46d3-a7fd-3a9ec341e2b4/volumes" Nov 03 21:12:14.867419 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:14.867376 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:15.127085 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:15.127050 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:15.627707 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:15.627667 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:16.330987 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:16.330889 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:17.104669 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:17.104634 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:17.531099 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:12:17.531058 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:12:17.605753 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:17.605714 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:17.907692 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:17.907600 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:18.605278 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:18.605246 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:19.313300 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.313268 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:19.448094 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448055 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47"] Nov 03 21:12:19.448585 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448565 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6efc79b0-e24f-4a4c-bf38-f291e291abb2" containerName="maas-api" Nov 03 21:12:19.448694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448585 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6efc79b0-e24f-4a4c-bf38-f291e291abb2" containerName="maas-api" Nov 03 21:12:19.448694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448615 4711 cpu_manager.go:414] "containerMap: removing container" podUID="bd468fb4-63e9-4e0f-be2d-73de9b58f1ab" containerName="payload-pre-processing" Nov 03 21:12:19.448694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448623 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd468fb4-63e9-4e0f-be2d-73de9b58f1ab" containerName="payload-pre-processing" Nov 03 21:12:19.448694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448641 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f0c02244-2313-495c-a0e9-52d7a1b9636b" containerName="main" Nov 03 21:12:19.448694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448647 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c02244-2313-495c-a0e9-52d7a1b9636b" containerName="main" Nov 03 21:12:19.448694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448662 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f28d3409-07ce-4d0f-a0d6-0dcb62551922" containerName="payload-processing" Nov 03 21:12:19.448694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448669 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28d3409-07ce-4d0f-a0d6-0dcb62551922" containerName="payload-processing" Nov 03 21:12:19.449124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448801 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6efc79b0-e24f-4a4c-bf38-f291e291abb2" containerName="maas-api" Nov 03 21:12:19.449124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448818 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd468fb4-63e9-4e0f-be2d-73de9b58f1ab" containerName="payload-pre-processing" Nov 03 21:12:19.449124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448827 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c02244-2313-495c-a0e9-52d7a1b9636b" containerName="main" Nov 03 21:12:19.449124 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.448838 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28d3409-07ce-4d0f-a0d6-0dcb62551922" containerName="payload-processing" Nov 03 21:12:19.452301 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.452281 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.459419 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.459343 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:12:19.459419 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.459385 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:12:19.460413 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.460334 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:12:19.460413 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.460388 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:12:19.461228 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.461203 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47"] Nov 03 21:12:19.605130 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.605052 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.605130 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.605119 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g25pf\" (UniqueName: \"kubernetes.io/projected/a31d12a9-9e14-4c25-a968-ccb96fdca465-kube-api-access-g25pf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.605308 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.605193 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.705897 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.705848 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.706077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.705939 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g25pf\" (UniqueName: \"kubernetes.io/projected/a31d12a9-9e14-4c25-a968-ccb96fdca465-kube-api-access-g25pf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.706077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.705965 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.706615 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.706596 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.708218 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.708197 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.718465 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.718416 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g25pf\" (UniqueName: \"kubernetes.io/projected/a31d12a9-9e14-4c25-a968-ccb96fdca465-kube-api-access-g25pf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:19.767491 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:19.767466 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:20.042945 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:20.042913 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47"] Nov 03 21:12:20.044570 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:12:20.044539 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31d12a9_9e14_4c25_a968_ccb96fdca465.slice/crio-4dcc5e33e993650bcafa701a3f35c503a8ab23e990d9e1ee4d0785d3bd06c7aa WatchSource:0}: Error finding container 4dcc5e33e993650bcafa701a3f35c503a8ab23e990d9e1ee4d0785d3bd06c7aa: Status 404 returned error can't find the container with id 4dcc5e33e993650bcafa701a3f35c503a8ab23e990d9e1ee4d0785d3bd06c7aa Nov 03 21:12:20.111725 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:20.111695 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:20.195408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:20.195375 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" event={"ID":"a31d12a9-9e14-4c25-a968-ccb96fdca465","Type":"ContainerStarted","Data":"4dcc5e33e993650bcafa701a3f35c503a8ab23e990d9e1ee4d0785d3bd06c7aa"} Nov 03 21:12:20.817366 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:20.817333 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:21.202438 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:21.202345 4711 generic.go:356] "Generic (PLEG): container finished" podID="a31d12a9-9e14-4c25-a968-ccb96fdca465" containerID="1d637a3e5279d839c146dae86feb29d2e5d56296ca95462e139b326eb3537075" exitCode=0 Nov 03 21:12:21.202438 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:21.202410 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" event={"ID":"a31d12a9-9e14-4c25-a968-ccb96fdca465","Type":"ContainerDied","Data":"1d637a3e5279d839c146dae86feb29d2e5d56296ca95462e139b326eb3537075"} Nov 03 21:12:21.524563 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:21.524525 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:22.212879 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.212843 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:22.354654 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.354627 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:22.431757 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.431723 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-xks-ca\") pod \"a31d12a9-9e14-4c25-a968-ccb96fdca465\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " Nov 03 21:12:22.431938 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.431811 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31d12a9-9e14-4c25-a968-ccb96fdca465-kube-api-access-g25pf\" (UniqueName: \"kubernetes.io/projected/a31d12a9-9e14-4c25-a968-ccb96fdca465-kube-api-access-g25pf\") pod \"a31d12a9-9e14-4c25-a968-ccb96fdca465\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " Nov 03 21:12:22.431938 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.431882 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-openshift-ca\") pod \"a31d12a9-9e14-4c25-a968-ccb96fdca465\" (UID: \"a31d12a9-9e14-4c25-a968-ccb96fdca465\") " Nov 03 21:12:22.432260 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.432235 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-openshift-ca" pod "a31d12a9-9e14-4c25-a968-ccb96fdca465" (UID: "a31d12a9-9e14-4c25-a968-ccb96fdca465"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:22.433754 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.433730 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-xks-ca" pod "a31d12a9-9e14-4c25-a968-ccb96fdca465" (UID: "a31d12a9-9e14-4c25-a968-ccb96fdca465"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:12:22.433901 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.433881 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31d12a9-9e14-4c25-a968-ccb96fdca465-kube-api-access-g25pf" pod "a31d12a9-9e14-4c25-a968-ccb96fdca465" (UID: "a31d12a9-9e14-4c25-a968-ccb96fdca465"). InnerVolumeSpecName "kube-api-access-g25pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:22.532661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.532635 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:22.532661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.532660 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g25pf\" (UniqueName: \"kubernetes.io/projected/a31d12a9-9e14-4c25-a968-ccb96fdca465-kube-api-access-g25pf\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:22.532832 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.532672 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a31d12a9-9e14-4c25-a968-ccb96fdca465-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:22.997165 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:22.997081 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:23.214024 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:23.213991 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" Nov 03 21:12:23.214477 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:23.214031 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47" event={"ID":"a31d12a9-9e14-4c25-a968-ccb96fdca465","Type":"ContainerDied","Data":"4dcc5e33e993650bcafa701a3f35c503a8ab23e990d9e1ee4d0785d3bd06c7aa"} Nov 03 21:12:23.214477 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:23.214076 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dcc5e33e993650bcafa701a3f35c503a8ab23e990d9e1ee4d0785d3bd06c7aa" Nov 03 21:12:23.707667 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:23.707633 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:23.839904 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:23.839864 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:24.523529 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:24.523492 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:25.319185 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:25.319153 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:26.005828 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:26.005780 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:26.700635 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:26.700602 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:27.511307 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:27.511266 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:28.206959 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:28.206915 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:28.903226 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:28.903191 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:29.611594 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:29.611558 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:30.137225 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:30.137191 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:30.272511 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:30.272477 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:30.405341 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:30.405244 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:30.890919 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:30.890879 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:31.063538 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:31.063493 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:31.071377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:31.071345 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47"] Nov 03 21:12:31.074183 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:31.074155 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-debeb1-hbw47"] Nov 03 21:12:31.199604 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:31.199517 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:31.421893 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:31.421852 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g"] Nov 03 21:12:31.422164 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:31.422118 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" containerID="cri-o://0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b" gracePeriod=30 Nov 03 21:12:31.517726 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:31.517689 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:32.678684 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:32.678642 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31d12a9-9e14-4c25-a968-ccb96fdca465" path="/var/lib/kubelet/pods/a31d12a9-9e14-4c25-a968-ccb96fdca465/volumes" Nov 03 21:12:36.668616 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.668593 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:12:36.773064 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773033 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-kube-api-access-s8v24\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-kube-api-access-s8v24\") pod \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " Nov 03 21:12:36.773238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773088 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istiod-ca-cert\") pod \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " Nov 03 21:12:36.773238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773119 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-certs\") pod \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " Nov 03 21:12:36.773238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773136 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-credential-socket\") pod \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " Nov 03 21:12:36.773238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773154 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-socket\") pod \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " Nov 03 21:12:36.773238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773181 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-envoy\") pod \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " Nov 03 21:12:36.773238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773236 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-token\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-token\") pod \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " Nov 03 21:12:36.773607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773273 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-podinfo\") pod \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " Nov 03 21:12:36.773607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773293 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-data\") pod \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\" (UID: \"b50ea865-76cc-45f8-aa8d-1dd4ad300707\") " Nov 03 21:12:36.773607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773434 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-certs" pod "b50ea865-76cc-45f8-aa8d-1dd4ad300707" (UID: "b50ea865-76cc-45f8-aa8d-1dd4ad300707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:36.773607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773497 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-credential-socket" pod "b50ea865-76cc-45f8-aa8d-1dd4ad300707" (UID: "b50ea865-76cc-45f8-aa8d-1dd4ad300707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:36.773819 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773672 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istiod-ca-cert" pod "b50ea865-76cc-45f8-aa8d-1dd4ad300707" (UID: "b50ea865-76cc-45f8-aa8d-1dd4ad300707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:36.773819 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773771 4711 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istiod-ca-cert\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:36.773819 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773792 4711 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:36.773819 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773785 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-socket" pod "b50ea865-76cc-45f8-aa8d-1dd4ad300707" (UID: "b50ea865-76cc-45f8-aa8d-1dd4ad300707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:36.773819 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.773808 4711 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-credential-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:36.774078 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.774011 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-data" pod "b50ea865-76cc-45f8-aa8d-1dd4ad300707" (UID: "b50ea865-76cc-45f8-aa8d-1dd4ad300707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:36.775486 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.775465 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-token" pod "b50ea865-76cc-45f8-aa8d-1dd4ad300707" (UID: "b50ea865-76cc-45f8-aa8d-1dd4ad300707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:36.775567 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.775494 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-podinfo" pod "b50ea865-76cc-45f8-aa8d-1dd4ad300707" (UID: "b50ea865-76cc-45f8-aa8d-1dd4ad300707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:12:36.775725 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.775700 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-envoy" pod "b50ea865-76cc-45f8-aa8d-1dd4ad300707" (UID: "b50ea865-76cc-45f8-aa8d-1dd4ad300707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:12:36.775725 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.775700 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-kube-api-access-s8v24" pod "b50ea865-76cc-45f8-aa8d-1dd4ad300707" (UID: "b50ea865-76cc-45f8-aa8d-1dd4ad300707"). InnerVolumeSpecName "kube-api-access-s8v24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:36.874755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.874717 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8v24\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-kube-api-access-s8v24\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:36.874755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.874748 4711 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-workload-socket\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:36.874755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.874762 4711 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-envoy\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:36.874998 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.874774 4711 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-token\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:36.874998 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.874785 4711 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-podinfo\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:36.874998 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.874797 4711 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b50ea865-76cc-45f8-aa8d-1dd4ad300707-istio-data\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:36.891629 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:36.891586 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:37.289506 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.289441 4711 generic.go:356] "Generic (PLEG): container finished" podID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerID="0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b" exitCode=0 Nov 03 21:12:37.289698 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.289520 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" Nov 03 21:12:37.289698 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.289539 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" event={"ID":"b50ea865-76cc-45f8-aa8d-1dd4ad300707","Type":"ContainerDied","Data":"0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b"} Nov 03 21:12:37.289698 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.289592 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g" event={"ID":"b50ea865-76cc-45f8-aa8d-1dd4ad300707","Type":"ContainerDied","Data":"0baf6f09b3a1fdff3b0bb895a80b58f9167e37faf2dc719272afef479aff8f3f"} Nov 03 21:12:37.289698 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.289613 4711 scope.go:122] "RemoveContainer" containerID="0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b" Nov 03 21:12:37.299256 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.299240 4711 scope.go:122] "RemoveContainer" containerID="51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788" Nov 03 21:12:37.307778 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.307761 4711 scope.go:122] "RemoveContainer" containerID="0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b" Nov 03 21:12:37.308031 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:12:37.308007 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b\": container with ID starting with 0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b not found: ID does not exist" containerID="0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b" Nov 03 21:12:37.308113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.308042 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b"} err="failed to get container status \"0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b\": rpc error: code = NotFound desc = could not find container \"0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b\": container with ID starting with 0e2044259fec348ad051d36d186e441991ad4edfb4cab4e9723f06a5e6b4d72b not found: ID does not exist" Nov 03 21:12:37.308113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.308066 4711 scope.go:122] "RemoveContainer" containerID="51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788" Nov 03 21:12:37.308335 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:12:37.308316 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788\": container with ID starting with 51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788 not found: ID does not exist" containerID="51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788" Nov 03 21:12:37.308378 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.308342 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788"} err="failed to get container status \"51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788\": rpc error: code = NotFound desc = could not find container \"51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788\": container with ID starting with 51bf50665bb30160c502c36a4edbe55cd5218784a4c657c28659ddf2a7908788 not found: ID does not exist" Nov 03 21:12:37.312727 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.312705 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g"] Nov 03 21:12:37.316202 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:37.316182 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-debeb1-openshift-default-7c4bd87c69trz2g"] Nov 03 21:12:38.678755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:38.678725 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" path="/var/lib/kubelet/pods/b50ea865-76cc-45f8-aa8d-1dd4ad300707/volumes" Nov 03 21:12:40.321468 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:40.318437 4711 scope.go:122] "RemoveContainer" containerID="7a90cd80312553c20dab528e7309217d359a3c021e43b6dd0555c2aa29815f1a" Nov 03 21:12:47.535162 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:12:47.535113 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:12:52.500537 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:52.500498 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:12:54.034187 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:54.034145 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6"] Nov 03 21:12:54.036325 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:54.036293 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-xxvt6"] Nov 03 21:12:54.678775 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:12:54.678745 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f68d2914-726a-4816-99bd-cecf235f9ba6" path="/var/lib/kubelet/pods/f68d2914-726a-4816-99bd-cecf235f9ba6/volumes" Nov 03 21:13:00.393366 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:00.393329 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:01.646930 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:01.646893 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:09.360412 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.360365 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx"] Nov 03 21:13:09.361018 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.360995 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" Nov 03 21:13:09.361115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.361020 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" Nov 03 21:13:09.361115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.361043 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" Nov 03 21:13:09.361115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.361051 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" Nov 03 21:13:09.361115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.361075 4711 cpu_manager.go:414] "containerMap: removing container" podUID="a31d12a9-9e14-4c25-a968-ccb96fdca465" containerName="revoke-keys" Nov 03 21:13:09.361115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.361083 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31d12a9-9e14-4c25-a968-ccb96fdca465" containerName="revoke-keys" Nov 03 21:13:09.361373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.361220 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" Nov 03 21:13:09.361373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.361235 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50ea865-76cc-45f8-aa8d-1dd4ad300707" containerName="istio-proxy" Nov 03 21:13:09.361373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.361249 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31d12a9-9e14-4c25-a968-ccb96fdca465" containerName="revoke-keys" Nov 03 21:13:09.364888 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.364866 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.370244 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.370222 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:13:09.371050 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.371010 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:13:09.371160 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.371063 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:13:09.371679 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.371649 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:13:09.374374 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.374353 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx"] Nov 03 21:13:09.466501 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.466466 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4j9wx\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.466501 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.466510 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4j9wx\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.466771 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.466613 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wp6\" (UniqueName: \"kubernetes.io/projected/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-kube-api-access-q7wp6\") pod \"maas-api-revoke-sub-452b6d7b-4j9wx\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.568171 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.568136 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4j9wx\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.568357 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.568183 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4j9wx\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.568357 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.568231 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wp6\" (UniqueName: \"kubernetes.io/projected/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-kube-api-access-q7wp6\") pod \"maas-api-revoke-sub-452b6d7b-4j9wx\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.568876 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.568856 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4j9wx\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.570425 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.570406 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4j9wx\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.580804 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.580779 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wp6\" (UniqueName: \"kubernetes.io/projected/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-kube-api-access-q7wp6\") pod \"maas-api-revoke-sub-452b6d7b-4j9wx\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.680085 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.680005 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:09.955115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:09.955088 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx"] Nov 03 21:13:09.956966 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:13:09.956940 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d11a7a1_bda7_4499_82f4_ac7ce5cd9266.slice/crio-1ec68594c52a004b45abd48dbd7c4267831c0209e92efce73e080cbf3f82e50c WatchSource:0}: Error finding container 1ec68594c52a004b45abd48dbd7c4267831c0209e92efce73e080cbf3f82e50c: Status 404 returned error can't find the container with id 1ec68594c52a004b45abd48dbd7c4267831c0209e92efce73e080cbf3f82e50c Nov 03 21:13:10.517709 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:10.517669 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" event={"ID":"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266","Type":"ContainerStarted","Data":"540978b2c075440f5c2210ea1256ae3292815fb9823bb43992481b2cbc97786c"} Nov 03 21:13:10.518122 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:10.517726 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" event={"ID":"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266","Type":"ContainerStarted","Data":"1ec68594c52a004b45abd48dbd7c4267831c0209e92efce73e080cbf3f82e50c"} Nov 03 21:13:10.540079 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:10.539967 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" podStartSLOduration=1.539949511 podStartE2EDuration="1.539949511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:10.537339397 +0000 UTC m=+4293.987445441" watchObservedRunningTime="2026-11-03 21:13:10.539949511 +0000 UTC m=+4293.990055557" Nov 03 21:13:11.527897 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:11.527846 4711 generic.go:356] "Generic (PLEG): container finished" podID="8d11a7a1-bda7-4499-82f4-ac7ce5cd9266" containerID="540978b2c075440f5c2210ea1256ae3292815fb9823bb43992481b2cbc97786c" exitCode=0 Nov 03 21:13:11.528431 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:11.527915 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" event={"ID":"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266","Type":"ContainerDied","Data":"540978b2c075440f5c2210ea1256ae3292815fb9823bb43992481b2cbc97786c"} Nov 03 21:13:12.864623 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:12.864597 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:13.039377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.039326 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-kube-api-access-q7wp6\" (UniqueName: \"kubernetes.io/projected/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-kube-api-access-q7wp6\") pod \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " Nov 03 21:13:13.039717 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.039691 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-openshift-ca\") pod \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " Nov 03 21:13:13.040050 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.040027 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-xks-ca\") pod \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\" (UID: \"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266\") " Nov 03 21:13:13.041717 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.041673 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-openshift-ca" pod "8d11a7a1-bda7-4499-82f4-ac7ce5cd9266" (UID: "8d11a7a1-bda7-4499-82f4-ac7ce5cd9266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:13.044284 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.043803 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-kube-api-access-q7wp6" pod "8d11a7a1-bda7-4499-82f4-ac7ce5cd9266" (UID: "8d11a7a1-bda7-4499-82f4-ac7ce5cd9266"). InnerVolumeSpecName "kube-api-access-q7wp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:13.044550 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.044529 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-xks-ca" pod "8d11a7a1-bda7-4499-82f4-ac7ce5cd9266" (UID: "8d11a7a1-bda7-4499-82f4-ac7ce5cd9266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:13.144562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.143574 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:13.144562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.143614 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7wp6\" (UniqueName: \"kubernetes.io/projected/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-kube-api-access-q7wp6\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:13.144562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.143628 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:13.549379 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.549349 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" Nov 03 21:13:13.549735 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.549361 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx" event={"ID":"8d11a7a1-bda7-4499-82f4-ac7ce5cd9266","Type":"ContainerDied","Data":"1ec68594c52a004b45abd48dbd7c4267831c0209e92efce73e080cbf3f82e50c"} Nov 03 21:13:13.549982 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:13.549961 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ec68594c52a004b45abd48dbd7c4267831c0209e92efce73e080cbf3f82e50c" Nov 03 21:13:17.531502 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:13:17.531464 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:13:18.022987 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:18.022954 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p"] Nov 03 21:13:18.027361 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:18.027332 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-87d6p"] Nov 03 21:13:18.678212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:18.678178 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d308aa8-1d73-46ab-9267-867b214ac899" path="/var/lib/kubelet/pods/4d308aa8-1d73-46ab-9267-867b214ac899/volumes" Nov 03 21:13:20.299919 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:20.299880 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:22.525635 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:22.525575 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:13:26.337511 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.337436 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn"] Nov 03 21:13:26.338604 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.338576 4711 cpu_manager.go:414] "containerMap: removing container" podUID="8d11a7a1-bda7-4499-82f4-ac7ce5cd9266" containerName="revoke-keys" Nov 03 21:13:26.338604 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.338602 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d11a7a1-bda7-4499-82f4-ac7ce5cd9266" containerName="revoke-keys" Nov 03 21:13:26.338897 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.338862 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d11a7a1-bda7-4499-82f4-ac7ce5cd9266" containerName="revoke-keys" Nov 03 21:13:26.348996 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.348962 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.351129 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.351094 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn"] Nov 03 21:13:26.353556 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.353527 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:13:26.354663 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.354640 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:13:26.354826 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.354678 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:13:26.354928 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.354729 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:13:26.375889 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.375849 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvfv\" (UniqueName: \"kubernetes.io/projected/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-kube-api-access-slvfv\") pod \"maas-api-revoke-sub-3e31a506-nlrcn\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.376075 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.376014 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-nlrcn\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.376134 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.376105 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-nlrcn\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.477526 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.477481 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-nlrcn\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.477738 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.477594 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvfv\" (UniqueName: \"kubernetes.io/projected/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-kube-api-access-slvfv\") pod \"maas-api-revoke-sub-3e31a506-nlrcn\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.477738 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.477723 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-nlrcn\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.478232 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.478207 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-nlrcn\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.480853 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.480830 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-nlrcn\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.490098 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.490069 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvfv\" (UniqueName: \"kubernetes.io/projected/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-kube-api-access-slvfv\") pod \"maas-api-revoke-sub-3e31a506-nlrcn\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.668625 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.668510 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:26.967337 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:26.967299 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn"] Nov 03 21:13:26.968431 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:13:26.968405 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1350a7ac_64a4_4578_b1c6_9dbe9897d1d8.slice/crio-52190636f0a8ddba3ad96af45b481c4b7e9deeb5dc7319191f1985d81be69d21 WatchSource:0}: Error finding container 52190636f0a8ddba3ad96af45b481c4b7e9deeb5dc7319191f1985d81be69d21: Status 404 returned error can't find the container with id 52190636f0a8ddba3ad96af45b481c4b7e9deeb5dc7319191f1985d81be69d21 Nov 03 21:13:27.655006 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:27.654919 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" event={"ID":"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8","Type":"ContainerStarted","Data":"27af624eefb64092683639887d5a984798704ad9638f75176d258eeaf5a03498"} Nov 03 21:13:27.655006 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:27.654973 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" event={"ID":"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8","Type":"ContainerStarted","Data":"52190636f0a8ddba3ad96af45b481c4b7e9deeb5dc7319191f1985d81be69d21"} Nov 03 21:13:27.688470 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:27.683606 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" podStartSLOduration=1.6835872219999999 podStartE2EDuration="1.683587222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:27.680898578 +0000 UTC m=+4311.131004639" watchObservedRunningTime="2026-11-03 21:13:27.683587222 +0000 UTC m=+4311.133693270" Nov 03 21:13:28.745309 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:28.745250 4711 generic.go:356] "Generic (PLEG): container finished" podID="1350a7ac-64a4-4578-b1c6-9dbe9897d1d8" containerID="27af624eefb64092683639887d5a984798704ad9638f75176d258eeaf5a03498" exitCode=0 Nov 03 21:13:28.746102 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:28.746056 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" event={"ID":"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8","Type":"ContainerDied","Data":"27af624eefb64092683639887d5a984798704ad9638f75176d258eeaf5a03498"} Nov 03 21:13:29.902278 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:29.902251 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:30.038315 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.038286 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-xks-ca\") pod \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " Nov 03 21:13:30.038545 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.038326 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-openshift-ca\") pod \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " Nov 03 21:13:30.038545 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.038364 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-kube-api-access-slvfv\" (UniqueName: \"kubernetes.io/projected/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-kube-api-access-slvfv\") pod \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\" (UID: \"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8\") " Nov 03 21:13:30.038756 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.038729 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-openshift-ca" pod "1350a7ac-64a4-4578-b1c6-9dbe9897d1d8" (UID: "1350a7ac-64a4-4578-b1c6-9dbe9897d1d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:30.040475 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.040439 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-kube-api-access-slvfv" pod "1350a7ac-64a4-4578-b1c6-9dbe9897d1d8" (UID: "1350a7ac-64a4-4578-b1c6-9dbe9897d1d8"). InnerVolumeSpecName "kube-api-access-slvfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:30.040617 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.040595 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-xks-ca" pod "1350a7ac-64a4-4578-b1c6-9dbe9897d1d8" (UID: "1350a7ac-64a4-4578-b1c6-9dbe9897d1d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:30.140271 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.140234 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:30.140271 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.140267 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:30.140271 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.140278 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slvfv\" (UniqueName: \"kubernetes.io/projected/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8-kube-api-access-slvfv\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:30.758168 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.758137 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" Nov 03 21:13:30.758369 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.758165 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn" event={"ID":"1350a7ac-64a4-4578-b1c6-9dbe9897d1d8","Type":"ContainerDied","Data":"52190636f0a8ddba3ad96af45b481c4b7e9deeb5dc7319191f1985d81be69d21"} Nov 03 21:13:30.758369 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:30.758211 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52190636f0a8ddba3ad96af45b481c4b7e9deeb5dc7319191f1985d81be69d21" Nov 03 21:13:35.023201 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:35.023158 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw"] Nov 03 21:13:35.025972 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:35.025942 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7hwkw"] Nov 03 21:13:36.678755 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:36.678720 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5548f8bf-2ca9-4517-86d5-0933d3949df1" path="/var/lib/kubelet/pods/5548f8bf-2ca9-4517-86d5-0933d3949df1/volumes" Nov 03 21:13:40.453622 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:40.453586 4711 scope.go:122] "RemoveContainer" containerID="3b968c410f389e22451267ad729497f9122c41bb7b5af28cdecec0aac467cd82" Nov 03 21:13:44.490738 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.490696 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6"] Nov 03 21:13:44.491350 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.491327 4711 cpu_manager.go:414] "containerMap: removing container" podUID="1350a7ac-64a4-4578-b1c6-9dbe9897d1d8" containerName="revoke-keys" Nov 03 21:13:44.491486 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.491353 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1350a7ac-64a4-4578-b1c6-9dbe9897d1d8" containerName="revoke-keys" Nov 03 21:13:44.491566 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.491493 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="1350a7ac-64a4-4578-b1c6-9dbe9897d1d8" containerName="revoke-keys" Nov 03 21:13:44.494612 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.494589 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.500719 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.500691 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:13:44.501561 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.501523 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:13:44.501561 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.501522 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:13:44.501777 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.501592 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:13:44.505039 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.505010 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6"] Nov 03 21:13:44.550713 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.550673 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45llj\" (UniqueName: \"kubernetes.io/projected/4c680771-2bf8-4a04-8668-d3a490cb8cde-kube-api-access-45llj\") pod \"maas-api-revoke-sub-df7f52cb-2qdt6\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.550905 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.550746 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7f52cb-2qdt6\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.550905 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.550832 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7f52cb-2qdt6\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.651797 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.651764 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7f52cb-2qdt6\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.651979 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.651804 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7f52cb-2qdt6\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.651979 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.651925 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45llj\" (UniqueName: \"kubernetes.io/projected/4c680771-2bf8-4a04-8668-d3a490cb8cde-kube-api-access-45llj\") pod \"maas-api-revoke-sub-df7f52cb-2qdt6\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.652559 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.652533 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7f52cb-2qdt6\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.654243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.654221 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7f52cb-2qdt6\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.664566 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.664533 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45llj\" (UniqueName: \"kubernetes.io/projected/4c680771-2bf8-4a04-8668-d3a490cb8cde-kube-api-access-45llj\") pod \"maas-api-revoke-sub-df7f52cb-2qdt6\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:44.809949 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:44.809905 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:45.098465 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:45.098424 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6"] Nov 03 21:13:45.100484 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:13:45.100437 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c680771_2bf8_4a04_8668_d3a490cb8cde.slice/crio-7f299f676ae4436d12eb8ab09ccee47647987d050f2ec0ac90edeba2d3a2b1e3 WatchSource:0}: Error finding container 7f299f676ae4436d12eb8ab09ccee47647987d050f2ec0ac90edeba2d3a2b1e3: Status 404 returned error can't find the container with id 7f299f676ae4436d12eb8ab09ccee47647987d050f2ec0ac90edeba2d3a2b1e3 Nov 03 21:13:45.873352 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:45.873312 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" event={"ID":"4c680771-2bf8-4a04-8668-d3a490cb8cde","Type":"ContainerStarted","Data":"64085e155fc5c18dbc5ba767e61a82150de2f45e76539c86a0a55c1b6f22d493"} Nov 03 21:13:45.873788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:45.873378 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" event={"ID":"4c680771-2bf8-4a04-8668-d3a490cb8cde","Type":"ContainerStarted","Data":"7f299f676ae4436d12eb8ab09ccee47647987d050f2ec0ac90edeba2d3a2b1e3"} Nov 03 21:13:45.893304 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:45.893246 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" podStartSLOduration=1.893228815 podStartE2EDuration="1.893228815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:45.89186868 +0000 UTC m=+4329.341974728" watchObservedRunningTime="2026-11-03 21:13:45.893228815 +0000 UTC m=+4329.343334861" Nov 03 21:13:46.023129 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:46.023095 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7"] Nov 03 21:13:46.025254 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:46.025223 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cvwj7"] Nov 03 21:13:46.681257 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:46.681162 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abca83b-537a-407c-8b65-b114dbf9f1fb" path="/var/lib/kubelet/pods/5abca83b-537a-407c-8b65-b114dbf9f1fb/volumes" Nov 03 21:13:46.879973 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:46.879937 4711 generic.go:356] "Generic (PLEG): container finished" podID="4c680771-2bf8-4a04-8668-d3a490cb8cde" containerID="64085e155fc5c18dbc5ba767e61a82150de2f45e76539c86a0a55c1b6f22d493" exitCode=0 Nov 03 21:13:46.880381 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:46.880022 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" event={"ID":"4c680771-2bf8-4a04-8668-d3a490cb8cde","Type":"ContainerDied","Data":"64085e155fc5c18dbc5ba767e61a82150de2f45e76539c86a0a55c1b6f22d493"} Nov 03 21:13:47.532071 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:13:47.532036 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:13:48.024384 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.024357 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:48.086205 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.086167 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c680771-2bf8-4a04-8668-d3a490cb8cde-kube-api-access-45llj\" (UniqueName: \"kubernetes.io/projected/4c680771-2bf8-4a04-8668-d3a490cb8cde-kube-api-access-45llj\") pod \"4c680771-2bf8-4a04-8668-d3a490cb8cde\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " Nov 03 21:13:48.086379 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.086282 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-openshift-ca\") pod \"4c680771-2bf8-4a04-8668-d3a490cb8cde\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " Nov 03 21:13:48.086379 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.086322 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-xks-ca\") pod \"4c680771-2bf8-4a04-8668-d3a490cb8cde\" (UID: \"4c680771-2bf8-4a04-8668-d3a490cb8cde\") " Nov 03 21:13:48.086801 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.086764 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-openshift-ca" pod "4c680771-2bf8-4a04-8668-d3a490cb8cde" (UID: "4c680771-2bf8-4a04-8668-d3a490cb8cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:48.088276 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.088252 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c680771-2bf8-4a04-8668-d3a490cb8cde-kube-api-access-45llj" pod "4c680771-2bf8-4a04-8668-d3a490cb8cde" (UID: "4c680771-2bf8-4a04-8668-d3a490cb8cde"). InnerVolumeSpecName "kube-api-access-45llj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:48.088640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.088619 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-xks-ca" pod "4c680771-2bf8-4a04-8668-d3a490cb8cde" (UID: "4c680771-2bf8-4a04-8668-d3a490cb8cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:48.187633 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.187557 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45llj\" (UniqueName: \"kubernetes.io/projected/4c680771-2bf8-4a04-8668-d3a490cb8cde-kube-api-access-45llj\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.187633 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.187586 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.187633 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.187596 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c680771-2bf8-4a04-8668-d3a490cb8cde-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:48.892680 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.892644 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" Nov 03 21:13:48.892680 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.892672 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6" event={"ID":"4c680771-2bf8-4a04-8668-d3a490cb8cde","Type":"ContainerDied","Data":"7f299f676ae4436d12eb8ab09ccee47647987d050f2ec0ac90edeba2d3a2b1e3"} Nov 03 21:13:48.892958 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:13:48.892725 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f299f676ae4436d12eb8ab09ccee47647987d050f2ec0ac90edeba2d3a2b1e3" Nov 03 21:14:00.466644 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:00.466594 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:01.707066 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:01.707027 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:04.023712 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:04.023670 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv"] Nov 03 21:14:04.028308 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:04.028274 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36ce3d9c-f8snv"] Nov 03 21:14:04.678839 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:04.678805 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05adac70-8ddc-4f3a-9bb9-138b3bef5193" path="/var/lib/kubelet/pods/05adac70-8ddc-4f3a-9bb9-138b3bef5193/volumes" Nov 03 21:14:10.028300 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:10.028264 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5"] Nov 03 21:14:15.028761 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:15.028707 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" podUID="5a13a60c-a9dd-4698-af2b-b6f139c83118" containerName="payload-pre-processing" containerID="cri-o://03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5" gracePeriod=25 Nov 03 21:14:15.286836 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:15.286770 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 21:14:15.464060 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:15.464015 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a13a60c-a9dd-4698-af2b-b6f139c83118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a13a60c-a9dd-4698-af2b-b6f139c83118-plugins-config-volume\") pod \"5a13a60c-a9dd-4698-af2b-b6f139c83118\" (UID: \"5a13a60c-a9dd-4698-af2b-b6f139c83118\") " Nov 03 21:14:15.464275 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:15.464137 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a13a60c-a9dd-4698-af2b-b6f139c83118-kube-api-access-xg5pr\" (UniqueName: \"kubernetes.io/projected/5a13a60c-a9dd-4698-af2b-b6f139c83118-kube-api-access-xg5pr\") pod \"5a13a60c-a9dd-4698-af2b-b6f139c83118\" (UID: \"5a13a60c-a9dd-4698-af2b-b6f139c83118\") " Nov 03 21:14:15.464397 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:15.464374 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a13a60c-a9dd-4698-af2b-b6f139c83118-plugins-config-volume" pod "5a13a60c-a9dd-4698-af2b-b6f139c83118" (UID: "5a13a60c-a9dd-4698-af2b-b6f139c83118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:15.464554 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:15.464540 4711 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a13a60c-a9dd-4698-af2b-b6f139c83118-plugins-config-volume\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:15.466303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:15.466278 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a13a60c-a9dd-4698-af2b-b6f139c83118-kube-api-access-xg5pr" pod "5a13a60c-a9dd-4698-af2b-b6f139c83118" (UID: "5a13a60c-a9dd-4698-af2b-b6f139c83118"). InnerVolumeSpecName "kube-api-access-xg5pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:15.565857 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:15.565773 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5pr\" (UniqueName: \"kubernetes.io/projected/5a13a60c-a9dd-4698-af2b-b6f139c83118-kube-api-access-xg5pr\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:16.039355 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.039317 4711 generic.go:356] "Generic (PLEG): container finished" podID="5a13a60c-a9dd-4698-af2b-b6f139c83118" containerID="03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5" exitCode=0 Nov 03 21:14:16.039783 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.039386 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" Nov 03 21:14:16.039783 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.039389 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" event={"ID":"5a13a60c-a9dd-4698-af2b-b6f139c83118","Type":"ContainerDied","Data":"03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5"} Nov 03 21:14:16.039783 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.039497 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5" event={"ID":"5a13a60c-a9dd-4698-af2b-b6f139c83118","Type":"ContainerDied","Data":"0f5f7e6971370a275d6eb2f6d0e4b642e6201624dcf43108457fcafcbc82f0e5"} Nov 03 21:14:16.039783 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.039530 4711 scope.go:122] "RemoveContainer" containerID="03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5" Nov 03 21:14:16.048372 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.048354 4711 scope.go:122] "RemoveContainer" containerID="03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5" Nov 03 21:14:16.048682 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:14:16.048662 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5\": container with ID starting with 03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5 not found: ID does not exist" containerID="03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5" Nov 03 21:14:16.048750 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.048691 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5"} err="failed to get container status \"03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5\": rpc error: code = NotFound desc = could not find container \"03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5\": container with ID starting with 03fd358eb709e1c530630d3623130fcf19a413e53ca9d0fe79db79e4649fbea5 not found: ID does not exist" Nov 03 21:14:16.062359 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.062334 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5"] Nov 03 21:14:16.066718 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.066692 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-7rjf5"] Nov 03 21:14:16.678829 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:16.678794 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a13a60c-a9dd-4698-af2b-b6f139c83118" path="/var/lib/kubelet/pods/5a13a60c-a9dd-4698-af2b-b6f139c83118/volumes" Nov 03 21:14:17.532238 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:14:17.532188 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:14:33.957215 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:33.957174 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:34.084266 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:34.084226 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:40.483208 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:40.483177 4711 scope.go:122] "RemoveContainer" containerID="7e589f6fb4d47ce0fb3dadb0f4d9eb20260b136d21afb93892ac622d632ef3f7" Nov 03 21:14:40.491065 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:40.491031 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:40.492458 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:40.492429 4711 scope.go:122] "RemoveContainer" containerID="265bc6a4e58672e6e5effcb08f91213df396a5ce204d1d159f13eca16a51dc45" Nov 03 21:14:40.500252 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:40.500235 4711 scope.go:122] "RemoveContainer" containerID="c351bbf1e7892c372c528b752466676b4b9d2d7817ce2a40f8c3ae9ed610af78" Nov 03 21:14:40.648352 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:40.648304 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:40.862870 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:40.862826 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:45.744633 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:45.744590 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:14:45.745046 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:45.744682 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:14:46.565669 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:14:46.565629 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:14:47.533507 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:14:47.533468 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:15:00.157757 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.157717 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2"] Nov 03 21:15:00.158182 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.158168 4711 cpu_manager.go:414] "containerMap: removing container" podUID="4c680771-2bf8-4a04-8668-d3a490cb8cde" containerName="revoke-keys" Nov 03 21:15:00.158229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.158183 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c680771-2bf8-4a04-8668-d3a490cb8cde" containerName="revoke-keys" Nov 03 21:15:00.158229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.158207 4711 cpu_manager.go:414] "containerMap: removing container" podUID="5a13a60c-a9dd-4698-af2b-b6f139c83118" containerName="payload-pre-processing" Nov 03 21:15:00.158229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.158212 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a13a60c-a9dd-4698-af2b-b6f139c83118" containerName="payload-pre-processing" Nov 03 21:15:00.158331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.158283 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c680771-2bf8-4a04-8668-d3a490cb8cde" containerName="revoke-keys" Nov 03 21:15:00.158331 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.158293 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a13a60c-a9dd-4698-af2b-b6f139c83118" containerName="payload-pre-processing" Nov 03 21:15:00.161673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.161657 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:00.169632 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.169599 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:15:00.169814 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.169627 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:15:00.169907 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.169746 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:15:00.170388 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.170362 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2"] Nov 03 21:15:00.278222 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.278189 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea403848-6e55-41e7-8e60-31d1459abc5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-whkw2\" (UID: \"ea403848-6e55-41e7-8e60-31d1459abc5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:00.278222 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.278224 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kgpq\" (UniqueName: \"kubernetes.io/projected/ea403848-6e55-41e7-8e60-31d1459abc5b-kube-api-access-5kgpq\") pod \"maas-api-key-cleanup-29895675-whkw2\" (UID: \"ea403848-6e55-41e7-8e60-31d1459abc5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:00.379093 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.379050 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea403848-6e55-41e7-8e60-31d1459abc5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-whkw2\" (UID: \"ea403848-6e55-41e7-8e60-31d1459abc5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:00.379093 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.379095 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgpq\" (UniqueName: \"kubernetes.io/projected/ea403848-6e55-41e7-8e60-31d1459abc5b-kube-api-access-5kgpq\") pod \"maas-api-key-cleanup-29895675-whkw2\" (UID: \"ea403848-6e55-41e7-8e60-31d1459abc5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:00.379753 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.379732 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea403848-6e55-41e7-8e60-31d1459abc5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895675-whkw2\" (UID: \"ea403848-6e55-41e7-8e60-31d1459abc5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:00.397494 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.397432 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgpq\" (UniqueName: \"kubernetes.io/projected/ea403848-6e55-41e7-8e60-31d1459abc5b-kube-api-access-5kgpq\") pod \"maas-api-key-cleanup-29895675-whkw2\" (UID: \"ea403848-6e55-41e7-8e60-31d1459abc5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:00.477548 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.477467 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:00.495085 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.495060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:00.758006 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:00.757977 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2"] Nov 03 21:15:00.760008 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:15:00.759982 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea403848_6e55_41e7_8e60_31d1459abc5b.slice/crio-f6d08095db6db70e621364965e91955408b53b0cf7a6625f296ff007bc024078 WatchSource:0}: Error finding container f6d08095db6db70e621364965e91955408b53b0cf7a6625f296ff007bc024078: Status 404 returned error can't find the container with id f6d08095db6db70e621364965e91955408b53b0cf7a6625f296ff007bc024078 Nov 03 21:15:01.320920 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:01.320882 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" event={"ID":"ea403848-6e55-41e7-8e60-31d1459abc5b","Type":"ContainerStarted","Data":"430c44fa858bed08d9bfcd336efedd67a632bc850adabdaec7c3f895dd0e9ed6"} Nov 03 21:15:01.321342 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:01.320941 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" event={"ID":"ea403848-6e55-41e7-8e60-31d1459abc5b","Type":"ContainerStarted","Data":"f6d08095db6db70e621364965e91955408b53b0cf7a6625f296ff007bc024078"} Nov 03 21:15:01.342177 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:01.342083 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" podStartSLOduration=1.3420702279999999 podStartE2EDuration="1.342070228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:01.339966394 +0000 UTC m=+4404.790072442" watchObservedRunningTime="2026-11-03 21:15:01.342070228 +0000 UTC m=+4404.792176274" Nov 03 21:15:01.751154 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:01.751060 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:15:02.328197 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:02.328163 4711 generic.go:356] "Generic (PLEG): container finished" podID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerID="430c44fa858bed08d9bfcd336efedd67a632bc850adabdaec7c3f895dd0e9ed6" exitCode=60 Nov 03 21:15:02.328683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:02.328250 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" event={"ID":"ea403848-6e55-41e7-8e60-31d1459abc5b","Type":"ContainerDied","Data":"430c44fa858bed08d9bfcd336efedd67a632bc850adabdaec7c3f895dd0e9ed6"} Nov 03 21:15:02.328683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:02.328483 4711 scope.go:122] "RemoveContainer" containerID="430c44fa858bed08d9bfcd336efedd67a632bc850adabdaec7c3f895dd0e9ed6" Nov 03 21:15:03.335113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:03.335016 4711 generic.go:356] "Generic (PLEG): container finished" podID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerID="f24f4cec76a3d27ef0e75c62628030d035f8a3cf3a3d6abb0852c2f452c732c4" exitCode=60 Nov 03 21:15:03.335113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:03.335100 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" event={"ID":"ea403848-6e55-41e7-8e60-31d1459abc5b","Type":"ContainerDied","Data":"f24f4cec76a3d27ef0e75c62628030d035f8a3cf3a3d6abb0852c2f452c732c4"} Nov 03 21:15:03.335667 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:03.335185 4711 scope.go:122] "RemoveContainer" containerID="430c44fa858bed08d9bfcd336efedd67a632bc850adabdaec7c3f895dd0e9ed6" Nov 03 21:15:03.335667 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:03.335410 4711 scope.go:122] "RemoveContainer" containerID="f24f4cec76a3d27ef0e75c62628030d035f8a3cf3a3d6abb0852c2f452c732c4" Nov 03 21:15:03.335792 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:15:03.335688 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-whkw2_odh-ai-gateway-infra(ea403848-6e55-41e7-8e60-31d1459abc5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" Nov 03 21:15:04.342389 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:04.342358 4711 scope.go:122] "RemoveContainer" containerID="f24f4cec76a3d27ef0e75c62628030d035f8a3cf3a3d6abb0852c2f452c732c4" Nov 03 21:15:04.342888 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:15:04.342619 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-whkw2_odh-ai-gateway-infra(ea403848-6e55-41e7-8e60-31d1459abc5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" Nov 03 21:15:08.024779 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:08.024734 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb"] Nov 03 21:15:08.026643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:08.026617 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d7f08ad-fz5nb"] Nov 03 21:15:08.678554 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:08.678518 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7b168c-32e6-4726-b339-338cf4250813" path="/var/lib/kubelet/pods/3e7b168c-32e6-4726-b339-338cf4250813/volumes" Nov 03 21:15:17.530727 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:15:17.530630 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:15:17.672636 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:17.672606 4711 scope.go:122] "RemoveContainer" containerID="f24f4cec76a3d27ef0e75c62628030d035f8a3cf3a3d6abb0852c2f452c732c4" Nov 03 21:15:18.415247 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:18.415208 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" event={"ID":"ea403848-6e55-41e7-8e60-31d1459abc5b","Type":"ContainerStarted","Data":"fc0d89cc14524718a0656cb895d2cd919c14f20abf828c4294279509572ac352"} Nov 03 21:15:19.422212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:19.422177 4711 generic.go:356] "Generic (PLEG): container finished" podID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerID="fc0d89cc14524718a0656cb895d2cd919c14f20abf828c4294279509572ac352" exitCode=60 Nov 03 21:15:19.422736 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:19.422251 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" event={"ID":"ea403848-6e55-41e7-8e60-31d1459abc5b","Type":"ContainerDied","Data":"fc0d89cc14524718a0656cb895d2cd919c14f20abf828c4294279509572ac352"} Nov 03 21:15:19.422736 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:19.422329 4711 scope.go:122] "RemoveContainer" containerID="f24f4cec76a3d27ef0e75c62628030d035f8a3cf3a3d6abb0852c2f452c732c4" Nov 03 21:15:19.422861 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:19.422740 4711 scope.go:122] "RemoveContainer" containerID="fc0d89cc14524718a0656cb895d2cd919c14f20abf828c4294279509572ac352" Nov 03 21:15:19.423048 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:15:19.423025 4711 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895675-whkw2_odh-ai-gateway-infra(ea403848-6e55-41e7-8e60-31d1459abc5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" Nov 03 21:15:19.450507 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:19.450477 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2"] Nov 03 21:15:20.566347 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:20.566324 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:20.587185 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:20.587154 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea403848-6e55-41e7-8e60-31d1459abc5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea403848-6e55-41e7-8e60-31d1459abc5b-maas-api-service-ca\") pod \"ea403848-6e55-41e7-8e60-31d1459abc5b\" (UID: \"ea403848-6e55-41e7-8e60-31d1459abc5b\") " Nov 03 21:15:20.587345 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:20.587243 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea403848-6e55-41e7-8e60-31d1459abc5b-kube-api-access-5kgpq\" (UniqueName: \"kubernetes.io/projected/ea403848-6e55-41e7-8e60-31d1459abc5b-kube-api-access-5kgpq\") pod \"ea403848-6e55-41e7-8e60-31d1459abc5b\" (UID: \"ea403848-6e55-41e7-8e60-31d1459abc5b\") " Nov 03 21:15:20.587688 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:20.587628 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea403848-6e55-41e7-8e60-31d1459abc5b-maas-api-service-ca" pod "ea403848-6e55-41e7-8e60-31d1459abc5b" (UID: "ea403848-6e55-41e7-8e60-31d1459abc5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:20.589655 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:20.589631 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea403848-6e55-41e7-8e60-31d1459abc5b-kube-api-access-5kgpq" pod "ea403848-6e55-41e7-8e60-31d1459abc5b" (UID: "ea403848-6e55-41e7-8e60-31d1459abc5b"). InnerVolumeSpecName "kube-api-access-5kgpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:20.688058 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:20.688031 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea403848-6e55-41e7-8e60-31d1459abc5b-maas-api-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.688058 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:20.688057 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kgpq\" (UniqueName: \"kubernetes.io/projected/ea403848-6e55-41e7-8e60-31d1459abc5b-kube-api-access-5kgpq\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:21.443699 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:21.443667 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" Nov 03 21:15:21.443892 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:21.443665 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2" event={"ID":"ea403848-6e55-41e7-8e60-31d1459abc5b","Type":"ContainerDied","Data":"f6d08095db6db70e621364965e91955408b53b0cf7a6625f296ff007bc024078"} Nov 03 21:15:21.443892 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:21.443823 4711 scope.go:122] "RemoveContainer" containerID="fc0d89cc14524718a0656cb895d2cd919c14f20abf828c4294279509572ac352" Nov 03 21:15:21.462853 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:21.462826 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2"] Nov 03 21:15:21.471941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:21.471911 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895675-whkw2"] Nov 03 21:15:22.678032 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:22.677997 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" path="/var/lib/kubelet/pods/ea403848-6e55-41e7-8e60-31d1459abc5b/volumes" Nov 03 21:15:24.031651 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:24.031614 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc"] Nov 03 21:15:24.034187 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:24.034158 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e1b39fb-5dbvc"] Nov 03 21:15:24.678500 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:24.678469 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b" path="/var/lib/kubelet/pods/2f0b174c-8a9e-4b0c-a514-97a34aa4ec4b/volumes" Nov 03 21:15:40.538776 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:40.538741 4711 scope.go:122] "RemoveContainer" containerID="f87bbe5a2fc44085daaa31d0688a059b3d1ef99b709423f1a1c55013404e59a0" Nov 03 21:15:40.548394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:15:40.548372 4711 scope.go:122] "RemoveContainer" containerID="511125e033299459087456806b26e5bb34188e56347ef15f0323a16747cae511" Nov 03 21:15:47.530981 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:15:47.530942 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:16:00.543781 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:00.543743 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:01.802408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:01.802369 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:07.060924 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.060885 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6"] Nov 03 21:16:07.061355 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.061317 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerName="cleanup" Nov 03 21:16:07.061355 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.061329 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerName="cleanup" Nov 03 21:16:07.061355 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.061356 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerName="cleanup" Nov 03 21:16:07.061516 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.061361 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerName="cleanup" Nov 03 21:16:07.061516 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.061432 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerName="cleanup" Nov 03 21:16:07.061516 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.061461 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerName="cleanup" Nov 03 21:16:07.064770 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.064752 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.071086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.071059 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:16:07.072653 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.072630 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:16:07.072653 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.072643 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:16:07.073190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.073171 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:16:07.075206 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.075184 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6"] Nov 03 21:16:07.139903 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.139871 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-98zg6\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.140110 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.140026 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j72w\" (UniqueName: \"kubernetes.io/projected/76115a5f-4577-4d58-9cd5-7b24d406e2bc-kube-api-access-9j72w\") pod \"maas-api-revoke-sub-e2547622-98zg6\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.140181 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.140161 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-98zg6\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.240774 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.240718 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j72w\" (UniqueName: \"kubernetes.io/projected/76115a5f-4577-4d58-9cd5-7b24d406e2bc-kube-api-access-9j72w\") pod \"maas-api-revoke-sub-e2547622-98zg6\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.240965 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.240849 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-98zg6\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.240965 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.240880 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-98zg6\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.241679 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.241658 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-98zg6\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.243421 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.243392 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-98zg6\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.253668 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.253641 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j72w\" (UniqueName: \"kubernetes.io/projected/76115a5f-4577-4d58-9cd5-7b24d406e2bc-kube-api-access-9j72w\") pod \"maas-api-revoke-sub-e2547622-98zg6\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.380661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.380573 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:07.656659 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.656634 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6"] Nov 03 21:16:07.658320 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:16:07.658293 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76115a5f_4577_4d58_9cd5_7b24d406e2bc.slice/crio-0bc3a7d92b0cbdf57f226278b30c795b7557d27608732940d3ba251ade5e08d1 WatchSource:0}: Error finding container 0bc3a7d92b0cbdf57f226278b30c795b7557d27608732940d3ba251ade5e08d1: Status 404 returned error can't find the container with id 0bc3a7d92b0cbdf57f226278b30c795b7557d27608732940d3ba251ade5e08d1 Nov 03 21:16:07.675807 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:07.675783 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" event={"ID":"76115a5f-4577-4d58-9cd5-7b24d406e2bc","Type":"ContainerStarted","Data":"0bc3a7d92b0cbdf57f226278b30c795b7557d27608732940d3ba251ade5e08d1"} Nov 03 21:16:08.690539 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:08.690422 4711 generic.go:356] "Generic (PLEG): container finished" podID="76115a5f-4577-4d58-9cd5-7b24d406e2bc" containerID="bfda6f2b526b315d30661dc5e047c31be2c27ceac7407dbec9b73d5c8622a940" exitCode=0 Nov 03 21:16:08.690539 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:08.690478 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" event={"ID":"76115a5f-4577-4d58-9cd5-7b24d406e2bc","Type":"ContainerDied","Data":"bfda6f2b526b315d30661dc5e047c31be2c27ceac7407dbec9b73d5c8622a940"} Nov 03 21:16:09.829660 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.829636 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:09.867117 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.867093 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-openshift-ca\") pod \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " Nov 03 21:16:09.867257 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.867178 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-xks-ca\") pod \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " Nov 03 21:16:09.867257 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.867228 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76115a5f-4577-4d58-9cd5-7b24d406e2bc-kube-api-access-9j72w\" (UniqueName: \"kubernetes.io/projected/76115a5f-4577-4d58-9cd5-7b24d406e2bc-kube-api-access-9j72w\") pod \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\" (UID: \"76115a5f-4577-4d58-9cd5-7b24d406e2bc\") " Nov 03 21:16:09.867517 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.867457 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-openshift-ca" pod "76115a5f-4577-4d58-9cd5-7b24d406e2bc" (UID: "76115a5f-4577-4d58-9cd5-7b24d406e2bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:09.869495 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.869439 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-xks-ca" pod "76115a5f-4577-4d58-9cd5-7b24d406e2bc" (UID: "76115a5f-4577-4d58-9cd5-7b24d406e2bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:09.869605 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.869544 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76115a5f-4577-4d58-9cd5-7b24d406e2bc-kube-api-access-9j72w" pod "76115a5f-4577-4d58-9cd5-7b24d406e2bc" (UID: "76115a5f-4577-4d58-9cd5-7b24d406e2bc"). InnerVolumeSpecName "kube-api-access-9j72w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:09.968284 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.968175 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:09.968284 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.968224 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76115a5f-4577-4d58-9cd5-7b24d406e2bc-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:09.968284 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:09.968240 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j72w\" (UniqueName: \"kubernetes.io/projected/76115a5f-4577-4d58-9cd5-7b24d406e2bc-kube-api-access-9j72w\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:10.701003 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:10.700972 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" Nov 03 21:16:10.701143 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:10.701009 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6" event={"ID":"76115a5f-4577-4d58-9cd5-7b24d406e2bc","Type":"ContainerDied","Data":"0bc3a7d92b0cbdf57f226278b30c795b7557d27608732940d3ba251ade5e08d1"} Nov 03 21:16:10.701143 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:10.701068 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bc3a7d92b0cbdf57f226278b30c795b7557d27608732940d3ba251ade5e08d1" Nov 03 21:16:17.533083 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:16:17.533035 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:16:17.669031 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:17.668995 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:19.616853 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:19.616816 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:16:23.780928 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:23.780891 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:16:23.781469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:23.781306 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:16:25.325200 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.325142 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq"] Nov 03 21:16:25.325851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.325834 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerName="cleanup" Nov 03 21:16:25.325944 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.325853 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerName="cleanup" Nov 03 21:16:25.325944 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.325865 4711 cpu_manager.go:414] "containerMap: removing container" podUID="76115a5f-4577-4d58-9cd5-7b24d406e2bc" containerName="revoke-keys" Nov 03 21:16:25.325944 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.325873 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76115a5f-4577-4d58-9cd5-7b24d406e2bc" containerName="revoke-keys" Nov 03 21:16:25.326119 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.325985 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea403848-6e55-41e7-8e60-31d1459abc5b" containerName="cleanup" Nov 03 21:16:25.326119 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.325997 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="76115a5f-4577-4d58-9cd5-7b24d406e2bc" containerName="revoke-keys" Nov 03 21:16:25.329645 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.329613 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.336709 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.336650 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:16:25.336959 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.336933 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:16:25.337050 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.337038 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:16:25.337113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.337106 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:16:25.343083 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.343060 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq"] Nov 03 21:16:25.482876 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.482816 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bnmhq\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.483096 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.483010 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bnmhq\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.483096 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.483048 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4q4x\" (UniqueName: \"kubernetes.io/projected/366a155e-df8c-4ed0-b1e3-f74750776407-kube-api-access-h4q4x\") pod \"maas-api-revoke-sub-c1e271a9-bnmhq\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.584516 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.584404 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bnmhq\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.584516 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.584453 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4q4x\" (UniqueName: \"kubernetes.io/projected/366a155e-df8c-4ed0-b1e3-f74750776407-kube-api-access-h4q4x\") pod \"maas-api-revoke-sub-c1e271a9-bnmhq\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.584516 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.584508 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bnmhq\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.585320 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.585289 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bnmhq\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.586962 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.586937 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bnmhq\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.597225 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.597175 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4q4x\" (UniqueName: \"kubernetes.io/projected/366a155e-df8c-4ed0-b1e3-f74750776407-kube-api-access-h4q4x\") pod \"maas-api-revoke-sub-c1e271a9-bnmhq\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.649618 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.649579 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:25.950161 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:25.950126 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq"] Nov 03 21:16:25.952494 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:16:25.952418 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366a155e_df8c_4ed0_b1e3_f74750776407.slice/crio-91e7f2e19bc5caaf1aca196bea43e081d8f9974da2c417b7aa033c333c7c10b5 WatchSource:0}: Error finding container 91e7f2e19bc5caaf1aca196bea43e081d8f9974da2c417b7aa033c333c7c10b5: Status 404 returned error can't find the container with id 91e7f2e19bc5caaf1aca196bea43e081d8f9974da2c417b7aa033c333c7c10b5 Nov 03 21:16:26.815251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:26.815210 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" event={"ID":"366a155e-df8c-4ed0-b1e3-f74750776407","Type":"ContainerStarted","Data":"c39985f5fd99cb24a33ee5eacff9f26e93a435314e8f0e6c4b7804cbb5e7cdfe"} Nov 03 21:16:26.816734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:26.815882 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" event={"ID":"366a155e-df8c-4ed0-b1e3-f74750776407","Type":"ContainerStarted","Data":"91e7f2e19bc5caaf1aca196bea43e081d8f9974da2c417b7aa033c333c7c10b5"} Nov 03 21:16:26.842232 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:26.842166 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" podStartSLOduration=1.84213333 podStartE2EDuration="1.84213333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:26.838937713 +0000 UTC m=+4490.289043761" watchObservedRunningTime="2026-11-03 21:16:26.84213333 +0000 UTC m=+4490.292239377" Nov 03 21:16:27.852596 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:27.852557 4711 generic.go:356] "Generic (PLEG): container finished" podID="366a155e-df8c-4ed0-b1e3-f74750776407" containerID="c39985f5fd99cb24a33ee5eacff9f26e93a435314e8f0e6c4b7804cbb5e7cdfe" exitCode=0 Nov 03 21:16:27.853326 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:27.852642 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" event={"ID":"366a155e-df8c-4ed0-b1e3-f74750776407","Type":"ContainerDied","Data":"c39985f5fd99cb24a33ee5eacff9f26e93a435314e8f0e6c4b7804cbb5e7cdfe"} Nov 03 21:16:29.009908 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.009874 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:29.063685 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.063650 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-openshift-ca\") pod \"366a155e-df8c-4ed0-b1e3-f74750776407\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " Nov 03 21:16:29.063845 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.063826 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366a155e-df8c-4ed0-b1e3-f74750776407-kube-api-access-h4q4x\" (UniqueName: \"kubernetes.io/projected/366a155e-df8c-4ed0-b1e3-f74750776407-kube-api-access-h4q4x\") pod \"366a155e-df8c-4ed0-b1e3-f74750776407\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " Nov 03 21:16:29.063909 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.063875 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-xks-ca\") pod \"366a155e-df8c-4ed0-b1e3-f74750776407\" (UID: \"366a155e-df8c-4ed0-b1e3-f74750776407\") " Nov 03 21:16:29.064100 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.064078 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-openshift-ca" pod "366a155e-df8c-4ed0-b1e3-f74750776407" (UID: "366a155e-df8c-4ed0-b1e3-f74750776407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:29.064210 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.064197 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:29.066007 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.065981 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-xks-ca" pod "366a155e-df8c-4ed0-b1e3-f74750776407" (UID: "366a155e-df8c-4ed0-b1e3-f74750776407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:29.066007 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.065992 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366a155e-df8c-4ed0-b1e3-f74750776407-kube-api-access-h4q4x" pod "366a155e-df8c-4ed0-b1e3-f74750776407" (UID: "366a155e-df8c-4ed0-b1e3-f74750776407"). InnerVolumeSpecName "kube-api-access-h4q4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:29.164917 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.164833 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4q4x\" (UniqueName: \"kubernetes.io/projected/366a155e-df8c-4ed0-b1e3-f74750776407-kube-api-access-h4q4x\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:29.164917 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.164864 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a155e-df8c-4ed0-b1e3-f74750776407-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:29.867656 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.867623 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" event={"ID":"366a155e-df8c-4ed0-b1e3-f74750776407","Type":"ContainerDied","Data":"91e7f2e19bc5caaf1aca196bea43e081d8f9974da2c417b7aa033c333c7c10b5"} Nov 03 21:16:29.867843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.867670 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq" Nov 03 21:16:29.867843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:29.867673 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91e7f2e19bc5caaf1aca196bea43e081d8f9974da2c417b7aa033c333c7c10b5" Nov 03 21:16:37.958722 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:37.958694 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:16:37.961650 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:37.961629 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:16:38.001324 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:38.001297 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:16:38.004145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:38.004126 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:16:40.580429 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:40.580391 4711 scope.go:122] "RemoveContainer" containerID="7c469759ac9060e180afd5e9ae735b34bd4662b335abfa633774df854ea8ab12" Nov 03 21:16:40.646795 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:40.646766 4711 scope.go:122] "RemoveContainer" containerID="790938efaa3288762b38b3f4c851ef2c3ff81ae3a4c8425d5dcb8a8e74760ad5" Nov 03 21:16:42.560899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.560861 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k"] Nov 03 21:16:42.561396 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.561376 4711 cpu_manager.go:414] "containerMap: removing container" podUID="366a155e-df8c-4ed0-b1e3-f74750776407" containerName="revoke-keys" Nov 03 21:16:42.561441 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.561397 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366a155e-df8c-4ed0-b1e3-f74750776407" containerName="revoke-keys" Nov 03 21:16:42.561498 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.561490 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="366a155e-df8c-4ed0-b1e3-f74750776407" containerName="revoke-keys" Nov 03 21:16:42.566203 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.566177 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.570235 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.570213 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:16:42.570392 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.570238 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:16:42.570392 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.570304 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:16:42.570392 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.570324 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:16:42.574838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.574812 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k"] Nov 03 21:16:42.661006 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.660972 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7c002-297d-45fb-8229-c86310e88672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x8x9k\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.661181 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.661024 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7c002-297d-45fb-8229-c86310e88672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x8x9k\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.661181 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.661145 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9htz\" (UniqueName: \"kubernetes.io/projected/fda7c002-297d-45fb-8229-c86310e88672-kube-api-access-g9htz\") pod \"maas-api-revoke-sub-2e1f5b5f-x8x9k\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.761889 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.761848 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7c002-297d-45fb-8229-c86310e88672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x8x9k\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.762115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.761912 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9htz\" (UniqueName: \"kubernetes.io/projected/fda7c002-297d-45fb-8229-c86310e88672-kube-api-access-g9htz\") pod \"maas-api-revoke-sub-2e1f5b5f-x8x9k\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.762115 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.761998 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7c002-297d-45fb-8229-c86310e88672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x8x9k\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.762623 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.762597 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7c002-297d-45fb-8229-c86310e88672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x8x9k\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.764388 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.764365 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7c002-297d-45fb-8229-c86310e88672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-x8x9k\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.774265 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.774242 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9htz\" (UniqueName: \"kubernetes.io/projected/fda7c002-297d-45fb-8229-c86310e88672-kube-api-access-g9htz\") pod \"maas-api-revoke-sub-2e1f5b5f-x8x9k\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:42.881627 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:42.881529 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:43.157882 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:43.157857 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k"] Nov 03 21:16:43.159395 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:16:43.159364 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda7c002_297d_45fb_8229_c86310e88672.slice/crio-b56dc48c5af1c1a3abc83c95a42ad7d9fa5184e2b022409b291529ae699bf7fc WatchSource:0}: Error finding container b56dc48c5af1c1a3abc83c95a42ad7d9fa5184e2b022409b291529ae699bf7fc: Status 404 returned error can't find the container with id b56dc48c5af1c1a3abc83c95a42ad7d9fa5184e2b022409b291529ae699bf7fc Nov 03 21:16:43.962036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:43.961999 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" event={"ID":"fda7c002-297d-45fb-8229-c86310e88672","Type":"ContainerStarted","Data":"9453ec923f095ebe54918f1c0f050c956cfb6b51fcd24e29e6a9ba681f108a76"} Nov 03 21:16:43.962571 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:43.962058 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" event={"ID":"fda7c002-297d-45fb-8229-c86310e88672","Type":"ContainerStarted","Data":"b56dc48c5af1c1a3abc83c95a42ad7d9fa5184e2b022409b291529ae699bf7fc"} Nov 03 21:16:43.981044 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:43.980981 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" podStartSLOduration=1.9809671789999999 podStartE2EDuration="1.980967179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:43.980065321 +0000 UTC m=+4507.430171371" watchObservedRunningTime="2026-11-03 21:16:43.980967179 +0000 UTC m=+4507.431073257" Nov 03 21:16:44.969497 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:44.969458 4711 generic.go:356] "Generic (PLEG): container finished" podID="fda7c002-297d-45fb-8229-c86310e88672" containerID="9453ec923f095ebe54918f1c0f050c956cfb6b51fcd24e29e6a9ba681f108a76" exitCode=0 Nov 03 21:16:44.969899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:44.969535 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" event={"ID":"fda7c002-297d-45fb-8229-c86310e88672","Type":"ContainerDied","Data":"9453ec923f095ebe54918f1c0f050c956cfb6b51fcd24e29e6a9ba681f108a76"} Nov 03 21:16:46.114123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.114096 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:46.199752 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.199721 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda7c002-297d-45fb-8229-c86310e88672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7c002-297d-45fb-8229-c86310e88672-maas-api-openshift-ca\") pod \"fda7c002-297d-45fb-8229-c86310e88672\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " Nov 03 21:16:46.199925 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.199827 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda7c002-297d-45fb-8229-c86310e88672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7c002-297d-45fb-8229-c86310e88672-maas-api-xks-ca\") pod \"fda7c002-297d-45fb-8229-c86310e88672\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " Nov 03 21:16:46.200008 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.199963 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda7c002-297d-45fb-8229-c86310e88672-kube-api-access-g9htz\" (UniqueName: \"kubernetes.io/projected/fda7c002-297d-45fb-8229-c86310e88672-kube-api-access-g9htz\") pod \"fda7c002-297d-45fb-8229-c86310e88672\" (UID: \"fda7c002-297d-45fb-8229-c86310e88672\") " Nov 03 21:16:46.200070 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.200053 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda7c002-297d-45fb-8229-c86310e88672-maas-api-openshift-ca" pod "fda7c002-297d-45fb-8229-c86310e88672" (UID: "fda7c002-297d-45fb-8229-c86310e88672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:46.200291 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.200266 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda7c002-297d-45fb-8229-c86310e88672-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:46.201908 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.201888 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda7c002-297d-45fb-8229-c86310e88672-maas-api-xks-ca" pod "fda7c002-297d-45fb-8229-c86310e88672" (UID: "fda7c002-297d-45fb-8229-c86310e88672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:46.201999 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.201984 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda7c002-297d-45fb-8229-c86310e88672-kube-api-access-g9htz" pod "fda7c002-297d-45fb-8229-c86310e88672" (UID: "fda7c002-297d-45fb-8229-c86310e88672"). InnerVolumeSpecName "kube-api-access-g9htz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:46.301690 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.301661 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda7c002-297d-45fb-8229-c86310e88672-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:46.301690 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.301688 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9htz\" (UniqueName: \"kubernetes.io/projected/fda7c002-297d-45fb-8229-c86310e88672-kube-api-access-g9htz\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:46.980740 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.980652 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" Nov 03 21:16:46.980899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.980650 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k" event={"ID":"fda7c002-297d-45fb-8229-c86310e88672","Type":"ContainerDied","Data":"b56dc48c5af1c1a3abc83c95a42ad7d9fa5184e2b022409b291529ae699bf7fc"} Nov 03 21:16:46.980899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:46.980778 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b56dc48c5af1c1a3abc83c95a42ad7d9fa5184e2b022409b291529ae699bf7fc" Nov 03 21:16:47.531460 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:16:47.531414 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:16:53.528433 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.528390 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz"] Nov 03 21:16:53.529119 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.529097 4711 cpu_manager.go:414] "containerMap: removing container" podUID="fda7c002-297d-45fb-8229-c86310e88672" containerName="revoke-keys" Nov 03 21:16:53.529210 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.529122 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda7c002-297d-45fb-8229-c86310e88672" containerName="revoke-keys" Nov 03 21:16:53.529277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.529244 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda7c002-297d-45fb-8229-c86310e88672" containerName="revoke-keys" Nov 03 21:16:53.532680 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.532651 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.538592 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.538393 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:16:53.539614 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.539587 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:16:53.539758 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.539642 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:16:53.539758 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.539682 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:16:53.541690 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.541664 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz"] Nov 03 21:16:53.576909 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.576876 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-x4gdz\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.577083 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.576924 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc86t\" (UniqueName: \"kubernetes.io/projected/81450211-1c91-4e25-9b2a-1f242ef56dd0-kube-api-access-dc86t\") pod \"maas-api-revoke-sub-ec843018-x4gdz\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.577083 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.577031 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-x4gdz\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.677569 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.677533 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-x4gdz\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.677765 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.677626 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-x4gdz\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.677765 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.677654 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc86t\" (UniqueName: \"kubernetes.io/projected/81450211-1c91-4e25-9b2a-1f242ef56dd0-kube-api-access-dc86t\") pod \"maas-api-revoke-sub-ec843018-x4gdz\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.678220 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.678196 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-x4gdz\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.679970 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.679951 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-x4gdz\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.690179 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.690156 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc86t\" (UniqueName: \"kubernetes.io/projected/81450211-1c91-4e25-9b2a-1f242ef56dd0-kube-api-access-dc86t\") pod \"maas-api-revoke-sub-ec843018-x4gdz\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:53.852691 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:53.852604 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:54.127053 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:54.126997 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz"] Nov 03 21:16:54.129610 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:16:54.129579 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81450211_1c91_4e25_9b2a_1f242ef56dd0.slice/crio-05ec076d80dc0bada8425cb52750bc906ba97a9f73785f5609ec16d864301947 WatchSource:0}: Error finding container 05ec076d80dc0bada8425cb52750bc906ba97a9f73785f5609ec16d864301947: Status 404 returned error can't find the container with id 05ec076d80dc0bada8425cb52750bc906ba97a9f73785f5609ec16d864301947 Nov 03 21:16:55.022436 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:55.022398 4711 generic.go:356] "Generic (PLEG): container finished" podID="81450211-1c91-4e25-9b2a-1f242ef56dd0" containerID="e02e14d105081d6239b117cc75286658d162412ec0f677b6ab2847f15b342745" exitCode=0 Nov 03 21:16:55.022886 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:55.022492 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" event={"ID":"81450211-1c91-4e25-9b2a-1f242ef56dd0","Type":"ContainerDied","Data":"e02e14d105081d6239b117cc75286658d162412ec0f677b6ab2847f15b342745"} Nov 03 21:16:55.022886 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:55.022556 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" event={"ID":"81450211-1c91-4e25-9b2a-1f242ef56dd0","Type":"ContainerStarted","Data":"05ec076d80dc0bada8425cb52750bc906ba97a9f73785f5609ec16d864301947"} Nov 03 21:16:56.207492 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.207457 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:16:56.304361 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.304258 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-openshift-ca\") pod \"81450211-1c91-4e25-9b2a-1f242ef56dd0\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " Nov 03 21:16:56.304361 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.304356 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-xks-ca\") pod \"81450211-1c91-4e25-9b2a-1f242ef56dd0\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " Nov 03 21:16:56.304661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.304393 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81450211-1c91-4e25-9b2a-1f242ef56dd0-kube-api-access-dc86t\" (UniqueName: \"kubernetes.io/projected/81450211-1c91-4e25-9b2a-1f242ef56dd0-kube-api-access-dc86t\") pod \"81450211-1c91-4e25-9b2a-1f242ef56dd0\" (UID: \"81450211-1c91-4e25-9b2a-1f242ef56dd0\") " Nov 03 21:16:56.304775 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.304752 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-openshift-ca" pod "81450211-1c91-4e25-9b2a-1f242ef56dd0" (UID: "81450211-1c91-4e25-9b2a-1f242ef56dd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:56.304895 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.304879 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:56.306893 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.306864 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81450211-1c91-4e25-9b2a-1f242ef56dd0-kube-api-access-dc86t" pod "81450211-1c91-4e25-9b2a-1f242ef56dd0" (UID: "81450211-1c91-4e25-9b2a-1f242ef56dd0"). InnerVolumeSpecName "kube-api-access-dc86t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:56.307083 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.307056 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-xks-ca" pod "81450211-1c91-4e25-9b2a-1f242ef56dd0" (UID: "81450211-1c91-4e25-9b2a-1f242ef56dd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:56.405894 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.405846 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81450211-1c91-4e25-9b2a-1f242ef56dd0-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:56.405894 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:56.405900 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc86t\" (UniqueName: \"kubernetes.io/projected/81450211-1c91-4e25-9b2a-1f242ef56dd0-kube-api-access-dc86t\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:57.043195 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:57.042406 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" event={"ID":"81450211-1c91-4e25-9b2a-1f242ef56dd0","Type":"ContainerDied","Data":"05ec076d80dc0bada8425cb52750bc906ba97a9f73785f5609ec16d864301947"} Nov 03 21:16:57.043195 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:57.042481 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05ec076d80dc0bada8425cb52750bc906ba97a9f73785f5609ec16d864301947" Nov 03 21:16:57.043195 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:16:57.042402 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz" Nov 03 21:17:00.601208 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:00.601170 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:01.862039 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:01.862002 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:17:17.535352 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:17:17.535312 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:17:19.552782 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.552751 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw"] Nov 03 21:17:19.553196 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.553182 4711 cpu_manager.go:414] "containerMap: removing container" podUID="81450211-1c91-4e25-9b2a-1f242ef56dd0" containerName="revoke-keys" Nov 03 21:17:19.553243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.553197 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81450211-1c91-4e25-9b2a-1f242ef56dd0" containerName="revoke-keys" Nov 03 21:17:19.553285 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.553266 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="81450211-1c91-4e25-9b2a-1f242ef56dd0" containerName="revoke-keys" Nov 03 21:17:19.556515 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.556488 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.564655 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.564070 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:17:19.564655 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.564158 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:17:19.564655 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.564204 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:17:19.564655 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.564259 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:17:19.565912 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.565885 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw"] Nov 03 21:17:19.680821 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.680783 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fsnjw\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.681376 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.681339 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspvk\" (UniqueName: \"kubernetes.io/projected/38eaf7ed-d304-4049-8d09-946fb484b688-kube-api-access-lspvk\") pod \"maas-api-revoke-sub-046e1343-fsnjw\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.681583 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.681556 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fsnjw\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.782982 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.782948 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fsnjw\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.782982 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.782984 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspvk\" (UniqueName: \"kubernetes.io/projected/38eaf7ed-d304-4049-8d09-946fb484b688-kube-api-access-lspvk\") pod \"maas-api-revoke-sub-046e1343-fsnjw\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.783273 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.783069 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fsnjw\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.783634 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.783606 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-fsnjw\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.785377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.785359 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-fsnjw\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.796108 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.796083 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspvk\" (UniqueName: \"kubernetes.io/projected/38eaf7ed-d304-4049-8d09-946fb484b688-kube-api-access-lspvk\") pod \"maas-api-revoke-sub-046e1343-fsnjw\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:19.872224 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:19.872130 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:20.148957 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:20.148929 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw"] Nov 03 21:17:20.151303 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:17:20.151279 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38eaf7ed_d304_4049_8d09_946fb484b688.slice/crio-aadcff4706c7eac5fb2c903e1d57b57188a68778e065aae8577dc372960b563a WatchSource:0}: Error finding container aadcff4706c7eac5fb2c903e1d57b57188a68778e065aae8577dc372960b563a: Status 404 returned error can't find the container with id aadcff4706c7eac5fb2c903e1d57b57188a68778e065aae8577dc372960b563a Nov 03 21:17:20.198894 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:20.198856 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" event={"ID":"38eaf7ed-d304-4049-8d09-946fb484b688","Type":"ContainerStarted","Data":"aadcff4706c7eac5fb2c903e1d57b57188a68778e065aae8577dc372960b563a"} Nov 03 21:17:21.205374 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:21.205341 4711 generic.go:356] "Generic (PLEG): container finished" podID="38eaf7ed-d304-4049-8d09-946fb484b688" containerID="aafc4d9b07fac03b8d66a4125c1301dcb5ee2da34a69ecca56e9b3a3d78794fb" exitCode=0 Nov 03 21:17:21.205775 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:21.205419 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" event={"ID":"38eaf7ed-d304-4049-8d09-946fb484b688","Type":"ContainerDied","Data":"aafc4d9b07fac03b8d66a4125c1301dcb5ee2da34a69ecca56e9b3a3d78794fb"} Nov 03 21:17:22.350142 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.350114 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:22.510643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.510610 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38eaf7ed-d304-4049-8d09-946fb484b688-kube-api-access-lspvk\" (UniqueName: \"kubernetes.io/projected/38eaf7ed-d304-4049-8d09-946fb484b688-kube-api-access-lspvk\") pod \"38eaf7ed-d304-4049-8d09-946fb484b688\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " Nov 03 21:17:22.510818 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.510657 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-openshift-ca\") pod \"38eaf7ed-d304-4049-8d09-946fb484b688\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " Nov 03 21:17:22.510818 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.510706 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-xks-ca\") pod \"38eaf7ed-d304-4049-8d09-946fb484b688\" (UID: \"38eaf7ed-d304-4049-8d09-946fb484b688\") " Nov 03 21:17:22.511036 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.511008 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-openshift-ca" pod "38eaf7ed-d304-4049-8d09-946fb484b688" (UID: "38eaf7ed-d304-4049-8d09-946fb484b688"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:22.512869 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.512843 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38eaf7ed-d304-4049-8d09-946fb484b688-kube-api-access-lspvk" pod "38eaf7ed-d304-4049-8d09-946fb484b688" (UID: "38eaf7ed-d304-4049-8d09-946fb484b688"). InnerVolumeSpecName "kube-api-access-lspvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:22.512935 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.512910 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-xks-ca" pod "38eaf7ed-d304-4049-8d09-946fb484b688" (UID: "38eaf7ed-d304-4049-8d09-946fb484b688"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:22.612003 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.611964 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lspvk\" (UniqueName: \"kubernetes.io/projected/38eaf7ed-d304-4049-8d09-946fb484b688-kube-api-access-lspvk\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:22.612003 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.612001 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:22.612214 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:22.612016 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38eaf7ed-d304-4049-8d09-946fb484b688-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:23.218125 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:23.218090 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" Nov 03 21:17:23.218289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:23.218122 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw" event={"ID":"38eaf7ed-d304-4049-8d09-946fb484b688","Type":"ContainerDied","Data":"aadcff4706c7eac5fb2c903e1d57b57188a68778e065aae8577dc372960b563a"} Nov 03 21:17:23.218289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:23.218171 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aadcff4706c7eac5fb2c903e1d57b57188a68778e065aae8577dc372960b563a" Nov 03 21:17:29.758223 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.758189 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq"] Nov 03 21:17:29.758729 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.758674 4711 cpu_manager.go:414] "containerMap: removing container" podUID="38eaf7ed-d304-4049-8d09-946fb484b688" containerName="revoke-keys" Nov 03 21:17:29.758729 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.758685 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38eaf7ed-d304-4049-8d09-946fb484b688" containerName="revoke-keys" Nov 03 21:17:29.758811 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.758749 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="38eaf7ed-d304-4049-8d09-946fb484b688" containerName="revoke-keys" Nov 03 21:17:29.762203 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.762182 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:29.767989 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.767826 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:17:29.767989 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.767887 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:17:29.767989 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.767955 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:17:29.768286 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.768196 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:17:29.771658 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.771092 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq"] Nov 03 21:17:29.883106 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.883063 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rbxvq\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:29.883302 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.883118 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdxr\" (UniqueName: \"kubernetes.io/projected/5d5c757f-b54b-4d46-9208-9341546e9aab-kube-api-access-vxdxr\") pod \"maas-api-revoke-sub-d3fba056-rbxvq\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:29.883351 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.883296 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rbxvq\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:29.984288 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.984247 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rbxvq\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:29.984523 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.984316 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rbxvq\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:29.984523 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.984345 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdxr\" (UniqueName: \"kubernetes.io/projected/5d5c757f-b54b-4d46-9208-9341546e9aab-kube-api-access-vxdxr\") pod \"maas-api-revoke-sub-d3fba056-rbxvq\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:29.984958 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.984932 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rbxvq\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:29.986780 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.986756 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rbxvq\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:29.996791 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:29.996769 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdxr\" (UniqueName: \"kubernetes.io/projected/5d5c757f-b54b-4d46-9208-9341546e9aab-kube-api-access-vxdxr\") pod \"maas-api-revoke-sub-d3fba056-rbxvq\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:30.079561 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:30.079430 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:30.356715 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:30.356683 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq"] Nov 03 21:17:30.358847 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:17:30.358815 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5c757f_b54b_4d46_9208_9341546e9aab.slice/crio-5ee1e1bcf7a10451dfd7f6a0ff44fa8dc81999d1e591f3f6299d3d261fc46a2f WatchSource:0}: Error finding container 5ee1e1bcf7a10451dfd7f6a0ff44fa8dc81999d1e591f3f6299d3d261fc46a2f: Status 404 returned error can't find the container with id 5ee1e1bcf7a10451dfd7f6a0ff44fa8dc81999d1e591f3f6299d3d261fc46a2f Nov 03 21:17:31.264687 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:31.264642 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" event={"ID":"5d5c757f-b54b-4d46-9208-9341546e9aab","Type":"ContainerStarted","Data":"d4e0eef1cd1c841a2025ef7785b0f3d042c7c57a969a1b7b9db8eee849196127"} Nov 03 21:17:31.265186 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:31.264716 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" event={"ID":"5d5c757f-b54b-4d46-9208-9341546e9aab","Type":"ContainerStarted","Data":"5ee1e1bcf7a10451dfd7f6a0ff44fa8dc81999d1e591f3f6299d3d261fc46a2f"} Nov 03 21:17:31.286184 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:31.286128 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" podStartSLOduration=2.286108733 podStartE2EDuration="2.286108733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:31.282899759 +0000 UTC m=+4554.733005805" watchObservedRunningTime="2026-11-03 21:17:31.286108733 +0000 UTC m=+4554.736214781" Nov 03 21:17:32.360022 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:32.359987 4711 generic.go:356] "Generic (PLEG): container finished" podID="5d5c757f-b54b-4d46-9208-9341546e9aab" containerID="d4e0eef1cd1c841a2025ef7785b0f3d042c7c57a969a1b7b9db8eee849196127" exitCode=0 Nov 03 21:17:32.360511 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:32.360056 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" event={"ID":"5d5c757f-b54b-4d46-9208-9341546e9aab","Type":"ContainerDied","Data":"d4e0eef1cd1c841a2025ef7785b0f3d042c7c57a969a1b7b9db8eee849196127"} Nov 03 21:17:33.725956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.725933 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:33.760758 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.760714 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-openshift-ca\") pod \"5d5c757f-b54b-4d46-9208-9341546e9aab\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " Nov 03 21:17:33.761464 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.761423 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5c757f-b54b-4d46-9208-9341546e9aab-kube-api-access-vxdxr\" (UniqueName: \"kubernetes.io/projected/5d5c757f-b54b-4d46-9208-9341546e9aab-kube-api-access-vxdxr\") pod \"5d5c757f-b54b-4d46-9208-9341546e9aab\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " Nov 03 21:17:33.761742 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.761723 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-xks-ca\") pod \"5d5c757f-b54b-4d46-9208-9341546e9aab\" (UID: \"5d5c757f-b54b-4d46-9208-9341546e9aab\") " Nov 03 21:17:33.762574 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.761326 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-openshift-ca" pod "5d5c757f-b54b-4d46-9208-9341546e9aab" (UID: "5d5c757f-b54b-4d46-9208-9341546e9aab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:33.765459 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.765416 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5c757f-b54b-4d46-9208-9341546e9aab-kube-api-access-vxdxr" pod "5d5c757f-b54b-4d46-9208-9341546e9aab" (UID: "5d5c757f-b54b-4d46-9208-9341546e9aab"). InnerVolumeSpecName "kube-api-access-vxdxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:33.765610 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.765421 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-xks-ca" pod "5d5c757f-b54b-4d46-9208-9341546e9aab" (UID: "5d5c757f-b54b-4d46-9208-9341546e9aab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:33.863748 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.863707 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:33.863748 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.863747 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d5c757f-b54b-4d46-9208-9341546e9aab-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:33.863748 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:33.863764 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdxr\" (UniqueName: \"kubernetes.io/projected/5d5c757f-b54b-4d46-9208-9341546e9aab-kube-api-access-vxdxr\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:34.370945 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:34.370913 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" Nov 03 21:17:34.371150 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:34.370959 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq" event={"ID":"5d5c757f-b54b-4d46-9208-9341546e9aab","Type":"ContainerDied","Data":"5ee1e1bcf7a10451dfd7f6a0ff44fa8dc81999d1e591f3f6299d3d261fc46a2f"} Nov 03 21:17:34.371150 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:34.371017 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ee1e1bcf7a10451dfd7f6a0ff44fa8dc81999d1e591f3f6299d3d261fc46a2f" Nov 03 21:17:45.741003 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:45.740966 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:17:45.741578 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:45.741062 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:17:47.533866 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:17:47.533828 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:17:50.632040 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.632005 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t"] Nov 03 21:17:50.632469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.632428 4711 cpu_manager.go:414] "containerMap: removing container" podUID="5d5c757f-b54b-4d46-9208-9341546e9aab" containerName="revoke-keys" Nov 03 21:17:50.632469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.632440 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5c757f-b54b-4d46-9208-9341546e9aab" containerName="revoke-keys" Nov 03 21:17:50.632587 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.632559 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5c757f-b54b-4d46-9208-9341546e9aab" containerName="revoke-keys" Nov 03 21:17:50.635638 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.635614 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.641822 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.641793 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:17:50.643273 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.643247 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:17:50.643837 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.643815 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:17:50.647031 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.647000 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t"] Nov 03 21:17:50.647184 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.647167 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:17:50.773771 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.773730 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-slf8t\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.773954 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.773783 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-slf8t\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.773954 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.773869 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdlrc\" (UniqueName: \"kubernetes.io/projected/6ece04e5-844c-43d5-91e2-00765459c610-kube-api-access-sdlrc\") pod \"maas-api-revoke-sub-ae594979-slf8t\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.875221 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.875183 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-slf8t\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.875441 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.875231 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdlrc\" (UniqueName: \"kubernetes.io/projected/6ece04e5-844c-43d5-91e2-00765459c610-kube-api-access-sdlrc\") pod \"maas-api-revoke-sub-ae594979-slf8t\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.875441 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.875400 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-slf8t\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.876080 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.876047 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-slf8t\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.877784 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.877752 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-slf8t\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.888244 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.888169 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdlrc\" (UniqueName: \"kubernetes.io/projected/6ece04e5-844c-43d5-91e2-00765459c610-kube-api-access-sdlrc\") pod \"maas-api-revoke-sub-ae594979-slf8t\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:50.952991 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:50.952957 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:51.233182 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:51.233154 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t"] Nov 03 21:17:51.235003 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:17:51.234967 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ece04e5_844c_43d5_91e2_00765459c610.slice/crio-f86498978d58acb93d37d475c68923ed50d7c8580e838541cbdceaf2335874a2 WatchSource:0}: Error finding container f86498978d58acb93d37d475c68923ed50d7c8580e838541cbdceaf2335874a2: Status 404 returned error can't find the container with id f86498978d58acb93d37d475c68923ed50d7c8580e838541cbdceaf2335874a2 Nov 03 21:17:51.477594 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:51.477559 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" event={"ID":"6ece04e5-844c-43d5-91e2-00765459c610","Type":"ContainerStarted","Data":"f86498978d58acb93d37d475c68923ed50d7c8580e838541cbdceaf2335874a2"} Nov 03 21:17:52.485216 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:52.485125 4711 generic.go:356] "Generic (PLEG): container finished" podID="6ece04e5-844c-43d5-91e2-00765459c610" containerID="f7c86d5f4dc4cece0502f147096b7517c30def81f6aedd889db02d91f0b86d4f" exitCode=0 Nov 03 21:17:52.485636 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:52.485211 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" event={"ID":"6ece04e5-844c-43d5-91e2-00765459c610","Type":"ContainerDied","Data":"f7c86d5f4dc4cece0502f147096b7517c30def81f6aedd889db02d91f0b86d4f"} Nov 03 21:17:53.630797 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.630770 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:53.709338 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.709288 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-xks-ca\") pod \"6ece04e5-844c-43d5-91e2-00765459c610\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " Nov 03 21:17:53.709552 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.709390 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ece04e5-844c-43d5-91e2-00765459c610-kube-api-access-sdlrc\" (UniqueName: \"kubernetes.io/projected/6ece04e5-844c-43d5-91e2-00765459c610-kube-api-access-sdlrc\") pod \"6ece04e5-844c-43d5-91e2-00765459c610\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " Nov 03 21:17:53.709552 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.709482 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-openshift-ca\") pod \"6ece04e5-844c-43d5-91e2-00765459c610\" (UID: \"6ece04e5-844c-43d5-91e2-00765459c610\") " Nov 03 21:17:53.709865 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.709841 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-openshift-ca" pod "6ece04e5-844c-43d5-91e2-00765459c610" (UID: "6ece04e5-844c-43d5-91e2-00765459c610"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:53.711439 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.711416 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-xks-ca" pod "6ece04e5-844c-43d5-91e2-00765459c610" (UID: "6ece04e5-844c-43d5-91e2-00765459c610"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:53.711553 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.711519 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ece04e5-844c-43d5-91e2-00765459c610-kube-api-access-sdlrc" pod "6ece04e5-844c-43d5-91e2-00765459c610" (UID: "6ece04e5-844c-43d5-91e2-00765459c610"). InnerVolumeSpecName "kube-api-access-sdlrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:53.811023 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.810990 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdlrc\" (UniqueName: \"kubernetes.io/projected/6ece04e5-844c-43d5-91e2-00765459c610-kube-api-access-sdlrc\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:53.811023 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.811020 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:53.811233 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:53.811030 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ece04e5-844c-43d5-91e2-00765459c610-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:54.496483 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:54.496454 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" Nov 03 21:17:54.496644 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:54.496486 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t" event={"ID":"6ece04e5-844c-43d5-91e2-00765459c610","Type":"ContainerDied","Data":"f86498978d58acb93d37d475c68923ed50d7c8580e838541cbdceaf2335874a2"} Nov 03 21:17:54.496644 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:17:54.496534 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f86498978d58acb93d37d475c68923ed50d7c8580e838541cbdceaf2335874a2" Nov 03 21:18:00.644212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:00.644173 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:01.914021 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:01.913990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:18:07.739022 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.738987 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7"] Nov 03 21:18:07.739473 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.739458 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6ece04e5-844c-43d5-91e2-00765459c610" containerName="revoke-keys" Nov 03 21:18:07.739473 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.739473 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ece04e5-844c-43d5-91e2-00765459c610" containerName="revoke-keys" Nov 03 21:18:07.739573 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.739536 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ece04e5-844c-43d5-91e2-00765459c610" containerName="revoke-keys" Nov 03 21:18:07.742957 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.742933 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:07.748186 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.748097 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:18:07.748313 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.748212 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:18:07.749792 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.749738 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:18:07.750795 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.750750 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:18:07.758168 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.758138 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7"] Nov 03 21:18:07.761327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.761300 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkqf7\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:07.762907 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.762883 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkqf7\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:07.763116 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.763091 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfs6c\" (UniqueName: \"kubernetes.io/projected/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-kube-api-access-dfs6c\") pod \"maas-api-revoke-sub-a4aef7d7-kkqf7\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:07.864222 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.864187 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkqf7\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:07.864222 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.864226 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkqf7\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:07.864479 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.864263 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfs6c\" (UniqueName: \"kubernetes.io/projected/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-kube-api-access-dfs6c\") pod \"maas-api-revoke-sub-a4aef7d7-kkqf7\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:07.864851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.864827 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkqf7\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:07.866586 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.866559 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-kkqf7\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:07.876633 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:07.876604 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfs6c\" (UniqueName: \"kubernetes.io/projected/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-kube-api-access-dfs6c\") pod \"maas-api-revoke-sub-a4aef7d7-kkqf7\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:08.060378 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:08.060347 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:08.336035 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:08.335957 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7"] Nov 03 21:18:08.339878 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:18:08.339850 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3b609b_31c4_46a1_a7c2_72b1ce7c7721.slice/crio-071379252f92176617024866398134c68bb7aec60320eaf4101f4f88fb257bec WatchSource:0}: Error finding container 071379252f92176617024866398134c68bb7aec60320eaf4101f4f88fb257bec: Status 404 returned error can't find the container with id 071379252f92176617024866398134c68bb7aec60320eaf4101f4f88fb257bec Nov 03 21:18:08.573099 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:08.573063 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" event={"ID":"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721","Type":"ContainerStarted","Data":"071379252f92176617024866398134c68bb7aec60320eaf4101f4f88fb257bec"} Nov 03 21:18:09.579489 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:09.579433 4711 generic.go:356] "Generic (PLEG): container finished" podID="0a3b609b-31c4-46a1-a7c2-72b1ce7c7721" containerID="b0d2cac2084f68cb1ff489caa998d3e467612716e36eb2d347257712243ef802" exitCode=0 Nov 03 21:18:09.579889 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:09.579539 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" event={"ID":"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721","Type":"ContainerDied","Data":"b0d2cac2084f68cb1ff489caa998d3e467612716e36eb2d347257712243ef802"} Nov 03 21:18:10.717860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.717834 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:10.792665 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.792636 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-openshift-ca\") pod \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " Nov 03 21:18:10.792846 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.792675 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-xks-ca\") pod \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " Nov 03 21:18:10.792846 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.792720 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-kube-api-access-dfs6c\" (UniqueName: \"kubernetes.io/projected/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-kube-api-access-dfs6c\") pod \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\" (UID: \"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721\") " Nov 03 21:18:10.793093 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.793058 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-openshift-ca" pod "0a3b609b-31c4-46a1-a7c2-72b1ce7c7721" (UID: "0a3b609b-31c4-46a1-a7c2-72b1ce7c7721"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:10.794894 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.794869 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-kube-api-access-dfs6c" pod "0a3b609b-31c4-46a1-a7c2-72b1ce7c7721" (UID: "0a3b609b-31c4-46a1-a7c2-72b1ce7c7721"). InnerVolumeSpecName "kube-api-access-dfs6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:10.794999 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.794926 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-xks-ca" pod "0a3b609b-31c4-46a1-a7c2-72b1ce7c7721" (UID: "0a3b609b-31c4-46a1-a7c2-72b1ce7c7721"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:10.894212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.894140 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:10.894212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.894166 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:10.894212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:10.894177 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfs6c\" (UniqueName: \"kubernetes.io/projected/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721-kube-api-access-dfs6c\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:11.592067 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:11.592038 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" Nov 03 21:18:11.592243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:11.592041 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7" event={"ID":"0a3b609b-31c4-46a1-a7c2-72b1ce7c7721","Type":"ContainerDied","Data":"071379252f92176617024866398134c68bb7aec60320eaf4101f4f88fb257bec"} Nov 03 21:18:11.592243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:11.592173 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="071379252f92176617024866398134c68bb7aec60320eaf4101f4f88fb257bec" Nov 03 21:18:13.022919 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:13.022881 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx"] Nov 03 21:18:13.025001 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:13.024981 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4j9wx"] Nov 03 21:18:14.678324 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:14.678291 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d11a7a1-bda7-4499-82f4-ac7ce5cd9266" path="/var/lib/kubelet/pods/8d11a7a1-bda7-4499-82f4-ac7ce5cd9266/volumes" Nov 03 21:18:17.531649 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:18:17.531576 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:18:23.246261 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.246216 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr"] Nov 03 21:18:23.247529 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.247502 4711 cpu_manager.go:414] "containerMap: removing container" podUID="0a3b609b-31c4-46a1-a7c2-72b1ce7c7721" containerName="revoke-keys" Nov 03 21:18:23.247674 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.247658 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3b609b-31c4-46a1-a7c2-72b1ce7c7721" containerName="revoke-keys" Nov 03 21:18:23.247956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.247938 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3b609b-31c4-46a1-a7c2-72b1ce7c7721" containerName="revoke-keys" Nov 03 21:18:23.252277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.252219 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.255759 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.255495 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr"] Nov 03 21:18:23.257951 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.257905 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:18:23.258286 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.258203 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:18:23.258286 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.258237 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:18:23.258557 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.258525 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:18:23.357180 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.357132 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rw7qr\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.357404 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.357204 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mp9\" (UniqueName: \"kubernetes.io/projected/2aaf955d-a69b-4136-bee0-ede342c65b8c-kube-api-access-s9mp9\") pod \"maas-api-revoke-sub-79805ca9-rw7qr\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.357404 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.357244 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rw7qr\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.458179 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.458147 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rw7qr\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.458385 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.458197 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mp9\" (UniqueName: \"kubernetes.io/projected/2aaf955d-a69b-4136-bee0-ede342c65b8c-kube-api-access-s9mp9\") pod \"maas-api-revoke-sub-79805ca9-rw7qr\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.458385 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.458230 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rw7qr\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.458996 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.458975 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rw7qr\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.460586 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.460564 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rw7qr\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.485361 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.485321 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mp9\" (UniqueName: \"kubernetes.io/projected/2aaf955d-a69b-4136-bee0-ede342c65b8c-kube-api-access-s9mp9\") pod \"maas-api-revoke-sub-79805ca9-rw7qr\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.630782 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.630738 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:23.829480 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.825545 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:18:23.829480 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.825615 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:18:23.850485 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.837049 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:18:23.850485 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.849753 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:18:23.986956 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:23.985814 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr"] Nov 03 21:18:23.986956 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:18:23.986881 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aaf955d_a69b_4136_bee0_ede342c65b8c.slice/crio-010a1438e844691e90ccfe57da32d8f7f7a33053fb906fa7e22b005620cee4ce WatchSource:0}: Error finding container 010a1438e844691e90ccfe57da32d8f7f7a33053fb906fa7e22b005620cee4ce: Status 404 returned error can't find the container with id 010a1438e844691e90ccfe57da32d8f7f7a33053fb906fa7e22b005620cee4ce Nov 03 21:18:24.663872 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:24.663771 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" event={"ID":"2aaf955d-a69b-4136-bee0-ede342c65b8c","Type":"ContainerStarted","Data":"753f446e859f6a4e56a4468231fdde50eac536025eece4cecd59ff9b5f362af1"} Nov 03 21:18:24.663872 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:24.663838 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" event={"ID":"2aaf955d-a69b-4136-bee0-ede342c65b8c","Type":"ContainerStarted","Data":"010a1438e844691e90ccfe57da32d8f7f7a33053fb906fa7e22b005620cee4ce"} Nov 03 21:18:24.683212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:24.683156 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" podStartSLOduration=1.6831419429999999 podStartE2EDuration="1.683141943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:18:24.681886468 +0000 UTC m=+4608.131992515" watchObservedRunningTime="2026-11-03 21:18:24.683141943 +0000 UTC m=+4608.133247989" Nov 03 21:18:25.669816 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:25.669775 4711 generic.go:356] "Generic (PLEG): container finished" podID="2aaf955d-a69b-4136-bee0-ede342c65b8c" containerID="753f446e859f6a4e56a4468231fdde50eac536025eece4cecd59ff9b5f362af1" exitCode=0 Nov 03 21:18:25.670216 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:25.669864 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" event={"ID":"2aaf955d-a69b-4136-bee0-ede342c65b8c","Type":"ContainerDied","Data":"753f446e859f6a4e56a4468231fdde50eac536025eece4cecd59ff9b5f362af1"} Nov 03 21:18:26.815378 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:26.815351 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:26.896260 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:26.896225 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aaf955d-a69b-4136-bee0-ede342c65b8c-kube-api-access-s9mp9\" (UniqueName: \"kubernetes.io/projected/2aaf955d-a69b-4136-bee0-ede342c65b8c-kube-api-access-s9mp9\") pod \"2aaf955d-a69b-4136-bee0-ede342c65b8c\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " Nov 03 21:18:26.896494 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:26.896319 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-openshift-ca\") pod \"2aaf955d-a69b-4136-bee0-ede342c65b8c\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " Nov 03 21:18:26.896728 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:26.896705 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-openshift-ca" pod "2aaf955d-a69b-4136-bee0-ede342c65b8c" (UID: "2aaf955d-a69b-4136-bee0-ede342c65b8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:26.898411 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:26.898391 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaf955d-a69b-4136-bee0-ede342c65b8c-kube-api-access-s9mp9" pod "2aaf955d-a69b-4136-bee0-ede342c65b8c" (UID: "2aaf955d-a69b-4136-bee0-ede342c65b8c"). InnerVolumeSpecName "kube-api-access-s9mp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:26.997054 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:26.996974 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-xks-ca\") pod \"2aaf955d-a69b-4136-bee0-ede342c65b8c\" (UID: \"2aaf955d-a69b-4136-bee0-ede342c65b8c\") " Nov 03 21:18:26.997238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:26.997206 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9mp9\" (UniqueName: \"kubernetes.io/projected/2aaf955d-a69b-4136-bee0-ede342c65b8c-kube-api-access-s9mp9\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:26.997238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:26.997219 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:26.999068 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:26.999035 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-xks-ca" pod "2aaf955d-a69b-4136-bee0-ede342c65b8c" (UID: "2aaf955d-a69b-4136-bee0-ede342c65b8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:27.098495 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:27.098436 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaf955d-a69b-4136-bee0-ede342c65b8c-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:27.681942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:27.681913 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" Nov 03 21:18:27.681942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:27.681928 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr" event={"ID":"2aaf955d-a69b-4136-bee0-ede342c65b8c","Type":"ContainerDied","Data":"010a1438e844691e90ccfe57da32d8f7f7a33053fb906fa7e22b005620cee4ce"} Nov 03 21:18:27.682166 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:27.681969 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="010a1438e844691e90ccfe57da32d8f7f7a33053fb906fa7e22b005620cee4ce" Nov 03 21:18:30.023015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:30.022974 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn"] Nov 03 21:18:30.026016 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:30.025989 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nlrcn"] Nov 03 21:18:30.678624 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:30.678588 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1350a7ac-64a4-4578-b1c6-9dbe9897d1d8" path="/var/lib/kubelet/pods/1350a7ac-64a4-4578-b1c6-9dbe9897d1d8/volumes" Nov 03 21:18:37.857707 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.857668 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z"] Nov 03 21:18:37.858298 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.858278 4711 cpu_manager.go:414] "containerMap: removing container" podUID="2aaf955d-a69b-4136-bee0-ede342c65b8c" containerName="revoke-keys" Nov 03 21:18:37.858346 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.858300 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aaf955d-a69b-4136-bee0-ede342c65b8c" containerName="revoke-keys" Nov 03 21:18:37.858499 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.858485 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aaf955d-a69b-4136-bee0-ede342c65b8c" containerName="revoke-keys" Nov 03 21:18:37.862399 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.862384 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:37.877550 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.867491 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:18:37.877550 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.868152 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:18:37.877550 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.868250 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:18:37.877550 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.870352 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z"] Nov 03 21:18:37.882885 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.878507 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:18:37.940469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.927055 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sp28z\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:37.940778 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.940740 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4qs2\" (UniqueName: \"kubernetes.io/projected/a375b334-ec91-4661-9bf6-03139a60920a-kube-api-access-k4qs2\") pod \"maas-api-revoke-sub-0b7e8edb-sp28z\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:37.941015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:37.940994 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sp28z\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:38.042509 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.042478 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sp28z\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:38.042797 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.042765 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4qs2\" (UniqueName: \"kubernetes.io/projected/a375b334-ec91-4661-9bf6-03139a60920a-kube-api-access-k4qs2\") pod \"maas-api-revoke-sub-0b7e8edb-sp28z\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:38.042902 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.042857 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sp28z\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:38.043800 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.043771 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sp28z\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:38.046586 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.045565 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sp28z\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:38.055048 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.055028 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4qs2\" (UniqueName: \"kubernetes.io/projected/a375b334-ec91-4661-9bf6-03139a60920a-kube-api-access-k4qs2\") pod \"maas-api-revoke-sub-0b7e8edb-sp28z\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:38.234629 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.234533 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:38.636468 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.633207 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z"] Nov 03 21:18:38.748677 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.748640 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" event={"ID":"a375b334-ec91-4661-9bf6-03139a60920a","Type":"ContainerStarted","Data":"fda5c0b2ae23979f71c760d3987c12edbe3350b7b2142193a2cd8b36b13915c7"} Nov 03 21:18:38.806469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.805571 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:18:38.806469 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.805667 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:18:38.829478 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.828556 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:18:38.829478 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:38.828625 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:18:39.755063 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:39.755030 4711 generic.go:356] "Generic (PLEG): container finished" podID="a375b334-ec91-4661-9bf6-03139a60920a" containerID="712ae9e3db951b34a0009061dbbcb333bbfdad3a8c191c1d9a63edd9d5e9d741" exitCode=0 Nov 03 21:18:39.755416 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:39.755120 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" event={"ID":"a375b334-ec91-4661-9bf6-03139a60920a","Type":"ContainerDied","Data":"712ae9e3db951b34a0009061dbbcb333bbfdad3a8c191c1d9a63edd9d5e9d741"} Nov 03 21:18:40.680327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:40.680293 4711 scope.go:122] "RemoveContainer" containerID="1d637a3e5279d839c146dae86feb29d2e5d56296ca95462e139b326eb3537075" Nov 03 21:18:40.894329 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:40.894302 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:40.974228 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:40.974146 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-xks-ca\") pod \"a375b334-ec91-4661-9bf6-03139a60920a\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " Nov 03 21:18:40.974394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:40.974235 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-openshift-ca\") pod \"a375b334-ec91-4661-9bf6-03139a60920a\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " Nov 03 21:18:40.974394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:40.974290 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a375b334-ec91-4661-9bf6-03139a60920a-kube-api-access-k4qs2\" (UniqueName: \"kubernetes.io/projected/a375b334-ec91-4661-9bf6-03139a60920a-kube-api-access-k4qs2\") pod \"a375b334-ec91-4661-9bf6-03139a60920a\" (UID: \"a375b334-ec91-4661-9bf6-03139a60920a\") " Nov 03 21:18:40.974643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:40.974619 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-openshift-ca" pod "a375b334-ec91-4661-9bf6-03139a60920a" (UID: "a375b334-ec91-4661-9bf6-03139a60920a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:40.976373 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:40.976348 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-xks-ca" pod "a375b334-ec91-4661-9bf6-03139a60920a" (UID: "a375b334-ec91-4661-9bf6-03139a60920a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:40.976531 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:40.976512 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a375b334-ec91-4661-9bf6-03139a60920a-kube-api-access-k4qs2" pod "a375b334-ec91-4661-9bf6-03139a60920a" (UID: "a375b334-ec91-4661-9bf6-03139a60920a"). InnerVolumeSpecName "kube-api-access-k4qs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:41.075510 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:41.075472 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:41.075510 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:41.075504 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a375b334-ec91-4661-9bf6-03139a60920a-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:41.075510 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:41.075516 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4qs2\" (UniqueName: \"kubernetes.io/projected/a375b334-ec91-4661-9bf6-03139a60920a-kube-api-access-k4qs2\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:41.766670 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:41.766640 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" Nov 03 21:18:41.766670 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:41.766664 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z" event={"ID":"a375b334-ec91-4661-9bf6-03139a60920a","Type":"ContainerDied","Data":"fda5c0b2ae23979f71c760d3987c12edbe3350b7b2142193a2cd8b36b13915c7"} Nov 03 21:18:41.766893 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:41.766706 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fda5c0b2ae23979f71c760d3987c12edbe3350b7b2142193a2cd8b36b13915c7" Nov 03 21:18:47.532975 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:18:47.532932 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:18:48.029062 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.029026 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6"] Nov 03 21:18:48.031477 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.031437 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7f52cb-2qdt6"] Nov 03 21:18:48.527732 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.527694 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk"] Nov 03 21:18:48.528198 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.528184 4711 cpu_manager.go:414] "containerMap: removing container" podUID="a375b334-ec91-4661-9bf6-03139a60920a" containerName="revoke-keys" Nov 03 21:18:48.528251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.528201 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a375b334-ec91-4661-9bf6-03139a60920a" containerName="revoke-keys" Nov 03 21:18:48.528305 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.528296 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="a375b334-ec91-4661-9bf6-03139a60920a" containerName="revoke-keys" Nov 03 21:18:48.531531 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.531513 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.538433 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.537998 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:18:48.538433 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.538000 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:18:48.538433 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.538077 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:18:48.538433 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.538137 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:18:48.542051 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.542026 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk"] Nov 03 21:18:48.650811 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.650780 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qftmk\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.650979 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.650854 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x878\" (UniqueName: \"kubernetes.io/projected/bae480ce-16d6-4e0f-83e7-0202970ba616-kube-api-access-7x878\") pod \"maas-api-revoke-sub-3d4156f1-qftmk\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.651027 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.651001 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qftmk\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.679132 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.679100 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c680771-2bf8-4a04-8668-d3a490cb8cde" path="/var/lib/kubelet/pods/4c680771-2bf8-4a04-8668-d3a490cb8cde/volumes" Nov 03 21:18:48.752424 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.752389 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qftmk\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.752608 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.752496 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x878\" (UniqueName: \"kubernetes.io/projected/bae480ce-16d6-4e0f-83e7-0202970ba616-kube-api-access-7x878\") pod \"maas-api-revoke-sub-3d4156f1-qftmk\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.752608 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.752569 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qftmk\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.753181 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.753146 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qftmk\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.754765 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.754739 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qftmk\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.764778 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.764751 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x878\" (UniqueName: \"kubernetes.io/projected/bae480ce-16d6-4e0f-83e7-0202970ba616-kube-api-access-7x878\") pod \"maas-api-revoke-sub-3d4156f1-qftmk\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:48.846362 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:48.846289 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:49.123116 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:49.123089 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk"] Nov 03 21:18:49.125619 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:18:49.125588 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae480ce_16d6_4e0f_83e7_0202970ba616.slice/crio-6c2b4296f9e4220676cd4721ea18348f273fe29ca4cfeb15df58415b9e7c38f5 WatchSource:0}: Error finding container 6c2b4296f9e4220676cd4721ea18348f273fe29ca4cfeb15df58415b9e7c38f5: Status 404 returned error can't find the container with id 6c2b4296f9e4220676cd4721ea18348f273fe29ca4cfeb15df58415b9e7c38f5 Nov 03 21:18:49.810134 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:49.810105 4711 generic.go:356] "Generic (PLEG): container finished" podID="bae480ce-16d6-4e0f-83e7-0202970ba616" containerID="7ecc696ecc5f4482b2ba8102304aa735b3fe757330e12b35b5f94c7d48a96fc6" exitCode=0 Nov 03 21:18:49.810579 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:49.810188 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" event={"ID":"bae480ce-16d6-4e0f-83e7-0202970ba616","Type":"ContainerDied","Data":"7ecc696ecc5f4482b2ba8102304aa735b3fe757330e12b35b5f94c7d48a96fc6"} Nov 03 21:18:49.810579 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:49.810251 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" event={"ID":"bae480ce-16d6-4e0f-83e7-0202970ba616","Type":"ContainerStarted","Data":"6c2b4296f9e4220676cd4721ea18348f273fe29ca4cfeb15df58415b9e7c38f5"} Nov 03 21:18:50.956089 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:50.956062 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:51.075636 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.075547 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-openshift-ca\") pod \"bae480ce-16d6-4e0f-83e7-0202970ba616\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " Nov 03 21:18:51.075636 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.075630 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-xks-ca\") pod \"bae480ce-16d6-4e0f-83e7-0202970ba616\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " Nov 03 21:18:51.075841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.075690 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae480ce-16d6-4e0f-83e7-0202970ba616-kube-api-access-7x878\" (UniqueName: \"kubernetes.io/projected/bae480ce-16d6-4e0f-83e7-0202970ba616-kube-api-access-7x878\") pod \"bae480ce-16d6-4e0f-83e7-0202970ba616\" (UID: \"bae480ce-16d6-4e0f-83e7-0202970ba616\") " Nov 03 21:18:51.076042 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.076014 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-openshift-ca" pod "bae480ce-16d6-4e0f-83e7-0202970ba616" (UID: "bae480ce-16d6-4e0f-83e7-0202970ba616"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:18:51.077772 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.077749 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-xks-ca" pod "bae480ce-16d6-4e0f-83e7-0202970ba616" (UID: "bae480ce-16d6-4e0f-83e7-0202970ba616"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:18:51.077918 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.077899 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae480ce-16d6-4e0f-83e7-0202970ba616-kube-api-access-7x878" pod "bae480ce-16d6-4e0f-83e7-0202970ba616" (UID: "bae480ce-16d6-4e0f-83e7-0202970ba616"). InnerVolumeSpecName "kube-api-access-7x878". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:18:51.176991 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.176959 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:51.176991 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.176986 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae480ce-16d6-4e0f-83e7-0202970ba616-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:51.176991 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.176995 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x878\" (UniqueName: \"kubernetes.io/projected/bae480ce-16d6-4e0f-83e7-0202970ba616-kube-api-access-7x878\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:18:51.822773 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.822737 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" event={"ID":"bae480ce-16d6-4e0f-83e7-0202970ba616","Type":"ContainerDied","Data":"6c2b4296f9e4220676cd4721ea18348f273fe29ca4cfeb15df58415b9e7c38f5"} Nov 03 21:18:51.822773 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.822758 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk" Nov 03 21:18:51.822990 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:51.822787 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c2b4296f9e4220676cd4721ea18348f273fe29ca4cfeb15df58415b9e7c38f5" Nov 03 21:18:59.007255 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:18:59.007215 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:00.313922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:00.313873 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:00.693781 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:00.693685 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:02.057751 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:02.057701 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:19:06.721877 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.721840 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk"] Nov 03 21:19:06.727077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.727045 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qftmk"] Nov 03 21:19:06.750466 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.750416 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l"] Nov 03 21:19:06.750880 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.750867 4711 cpu_manager.go:414] "containerMap: removing container" podUID="bae480ce-16d6-4e0f-83e7-0202970ba616" containerName="revoke-keys" Nov 03 21:19:06.750946 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.750882 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae480ce-16d6-4e0f-83e7-0202970ba616" containerName="revoke-keys" Nov 03 21:19:06.750999 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.750952 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae480ce-16d6-4e0f-83e7-0202970ba616" containerName="revoke-keys" Nov 03 21:19:06.754380 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.754362 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:06.759979 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.759957 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:19:06.760152 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.760081 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:19:06.760237 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.760108 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:19:06.761220 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.761198 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:19:06.769212 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.769182 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l"] Nov 03 21:19:06.890172 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.890137 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khzzc\" (UniqueName: \"kubernetes.io/projected/8afb598b-e19d-4f19-82ee-b55761f49a88-kube-api-access-khzzc\") pod \"maas-api-revoke-sub-3d4156f1-wjh7l\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:06.890377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.890191 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjh7l\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:06.890377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.890332 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjh7l\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:06.991785 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.991703 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjh7l\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:06.991967 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.991815 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzzc\" (UniqueName: \"kubernetes.io/projected/8afb598b-e19d-4f19-82ee-b55761f49a88-kube-api-access-khzzc\") pod \"maas-api-revoke-sub-3d4156f1-wjh7l\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:06.991967 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.991874 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjh7l\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:06.992549 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.992523 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjh7l\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:06.994021 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:06.993997 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjh7l\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:07.004030 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:07.004010 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khzzc\" (UniqueName: \"kubernetes.io/projected/8afb598b-e19d-4f19-82ee-b55761f49a88-kube-api-access-khzzc\") pod \"maas-api-revoke-sub-3d4156f1-wjh7l\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:07.069805 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:07.069774 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:07.345163 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:07.345137 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l"] Nov 03 21:19:07.347107 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:19:07.347080 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afb598b_e19d_4f19_82ee_b55761f49a88.slice/crio-679492035a836be6c1f2104ffa0c2c5137c8d9c3e58486fe845f50b1f19099b2 WatchSource:0}: Error finding container 679492035a836be6c1f2104ffa0c2c5137c8d9c3e58486fe845f50b1f19099b2: Status 404 returned error can't find the container with id 679492035a836be6c1f2104ffa0c2c5137c8d9c3e58486fe845f50b1f19099b2 Nov 03 21:19:07.959332 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:07.959245 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" event={"ID":"8afb598b-e19d-4f19-82ee-b55761f49a88","Type":"ContainerStarted","Data":"f39267bb0cce8a7b68faf28d7ca9eb36daf8692ba749a2276d6ddce9889b1841"} Nov 03 21:19:07.959332 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:07.959303 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" event={"ID":"8afb598b-e19d-4f19-82ee-b55761f49a88","Type":"ContainerStarted","Data":"679492035a836be6c1f2104ffa0c2c5137c8d9c3e58486fe845f50b1f19099b2"} Nov 03 21:19:07.979824 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:07.979774 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" podStartSLOduration=1.9797614000000001 podStartE2EDuration="1.9797614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:07.979205513 +0000 UTC m=+4651.429311572" watchObservedRunningTime="2026-11-03 21:19:07.9797614 +0000 UTC m=+4651.429867505" Nov 03 21:19:08.678895 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:08.678863 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae480ce-16d6-4e0f-83e7-0202970ba616" path="/var/lib/kubelet/pods/bae480ce-16d6-4e0f-83e7-0202970ba616/volumes" Nov 03 21:19:08.966553 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:08.966463 4711 generic.go:356] "Generic (PLEG): container finished" podID="8afb598b-e19d-4f19-82ee-b55761f49a88" containerID="f39267bb0cce8a7b68faf28d7ca9eb36daf8692ba749a2276d6ddce9889b1841" exitCode=0 Nov 03 21:19:08.966922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:08.966541 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" event={"ID":"8afb598b-e19d-4f19-82ee-b55761f49a88","Type":"ContainerDied","Data":"f39267bb0cce8a7b68faf28d7ca9eb36daf8692ba749a2276d6ddce9889b1841"} Nov 03 21:19:10.116567 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.116541 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:10.118013 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.117989 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-openshift-ca\") pod \"8afb598b-e19d-4f19-82ee-b55761f49a88\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " Nov 03 21:19:10.118106 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.118094 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-xks-ca\") pod \"8afb598b-e19d-4f19-82ee-b55761f49a88\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " Nov 03 21:19:10.118171 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.118147 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8afb598b-e19d-4f19-82ee-b55761f49a88-kube-api-access-khzzc\" (UniqueName: \"kubernetes.io/projected/8afb598b-e19d-4f19-82ee-b55761f49a88-kube-api-access-khzzc\") pod \"8afb598b-e19d-4f19-82ee-b55761f49a88\" (UID: \"8afb598b-e19d-4f19-82ee-b55761f49a88\") " Nov 03 21:19:10.118351 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.118329 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-openshift-ca" pod "8afb598b-e19d-4f19-82ee-b55761f49a88" (UID: "8afb598b-e19d-4f19-82ee-b55761f49a88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:10.118561 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.118542 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:10.120274 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.120251 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afb598b-e19d-4f19-82ee-b55761f49a88-kube-api-access-khzzc" pod "8afb598b-e19d-4f19-82ee-b55761f49a88" (UID: "8afb598b-e19d-4f19-82ee-b55761f49a88"). InnerVolumeSpecName "kube-api-access-khzzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:10.120363 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.120313 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-xks-ca" pod "8afb598b-e19d-4f19-82ee-b55761f49a88" (UID: "8afb598b-e19d-4f19-82ee-b55761f49a88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:19:10.219332 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.219302 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8afb598b-e19d-4f19-82ee-b55761f49a88-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:10.219332 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.219330 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khzzc\" (UniqueName: \"kubernetes.io/projected/8afb598b-e19d-4f19-82ee-b55761f49a88-kube-api-access-khzzc\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:10.979179 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.979095 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" Nov 03 21:19:10.979179 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.979121 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l" event={"ID":"8afb598b-e19d-4f19-82ee-b55761f49a88","Type":"ContainerDied","Data":"679492035a836be6c1f2104ffa0c2c5137c8d9c3e58486fe845f50b1f19099b2"} Nov 03 21:19:10.979179 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:10.979175 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="679492035a836be6c1f2104ffa0c2c5137c8d9c3e58486fe845f50b1f19099b2" Nov 03 21:19:17.532703 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:19:17.532658 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:19:30.736605 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:30.736549 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:19:30.737138 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:30.736959 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:19:35.364537 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.364504 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc"] Nov 03 21:19:35.364968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.364935 4711 cpu_manager.go:414] "containerMap: removing container" podUID="8afb598b-e19d-4f19-82ee-b55761f49a88" containerName="revoke-keys" Nov 03 21:19:35.364968 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.364947 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afb598b-e19d-4f19-82ee-b55761f49a88" containerName="revoke-keys" Nov 03 21:19:35.365061 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.365037 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afb598b-e19d-4f19-82ee-b55761f49a88" containerName="revoke-keys" Nov 03 21:19:35.368385 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.368357 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.373674 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.373649 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:19:35.374741 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.374720 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:19:35.374866 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.374783 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:19:35.375072 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.375055 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:19:35.379118 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.379096 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc"] Nov 03 21:19:35.389191 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.389141 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64pg\" (UniqueName: \"kubernetes.io/projected/a43ccea2-885d-4d95-82af-d3649027e604-kube-api-access-z64pg\") pod \"maas-api-revoke-sub-e2fc9c5b-q4ccc\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.389366 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.389233 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-q4ccc\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.389459 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.389424 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-q4ccc\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.490556 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.490525 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-q4ccc\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.490735 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.490583 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64pg\" (UniqueName: \"kubernetes.io/projected/a43ccea2-885d-4d95-82af-d3649027e604-kube-api-access-z64pg\") pod \"maas-api-revoke-sub-e2fc9c5b-q4ccc\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.490735 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.490690 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-q4ccc\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.491188 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.491166 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-q4ccc\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.492966 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.492945 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-q4ccc\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.502836 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.502814 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64pg\" (UniqueName: \"kubernetes.io/projected/a43ccea2-885d-4d95-82af-d3649027e604-kube-api-access-z64pg\") pod \"maas-api-revoke-sub-e2fc9c5b-q4ccc\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.684337 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.684266 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:35.959488 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:35.959457 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc"] Nov 03 21:19:35.961542 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:19:35.961507 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43ccea2_885d_4d95_82af_d3649027e604.slice/crio-15e84373ab9ff85d0117c3319aab63efaeb022d0a358071e41d0477f5b34217d WatchSource:0}: Error finding container 15e84373ab9ff85d0117c3319aab63efaeb022d0a358071e41d0477f5b34217d: Status 404 returned error can't find the container with id 15e84373ab9ff85d0117c3319aab63efaeb022d0a358071e41d0477f5b34217d Nov 03 21:19:36.144619 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:36.144582 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" event={"ID":"a43ccea2-885d-4d95-82af-d3649027e604","Type":"ContainerStarted","Data":"15e84373ab9ff85d0117c3319aab63efaeb022d0a358071e41d0477f5b34217d"} Nov 03 21:19:37.153762 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:37.153663 4711 generic.go:356] "Generic (PLEG): container finished" podID="a43ccea2-885d-4d95-82af-d3649027e604" containerID="e583be9166122802cf830e94b3a02fb801ee3f1e1ddd2dabf5c671bd805818c0" exitCode=0 Nov 03 21:19:37.154252 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:37.153909 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" event={"ID":"a43ccea2-885d-4d95-82af-d3649027e604","Type":"ContainerDied","Data":"e583be9166122802cf830e94b3a02fb801ee3f1e1ddd2dabf5c671bd805818c0"} Nov 03 21:19:38.303496 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.303469 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:38.420487 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.420357 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43ccea2-885d-4d95-82af-d3649027e604-kube-api-access-z64pg\" (UniqueName: \"kubernetes.io/projected/a43ccea2-885d-4d95-82af-d3649027e604-kube-api-access-z64pg\") pod \"a43ccea2-885d-4d95-82af-d3649027e604\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " Nov 03 21:19:38.420678 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.420504 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-xks-ca\") pod \"a43ccea2-885d-4d95-82af-d3649027e604\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " Nov 03 21:19:38.420678 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.420550 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-openshift-ca\") pod \"a43ccea2-885d-4d95-82af-d3649027e604\" (UID: \"a43ccea2-885d-4d95-82af-d3649027e604\") " Nov 03 21:19:38.420941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.420914 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-openshift-ca" pod "a43ccea2-885d-4d95-82af-d3649027e604" (UID: "a43ccea2-885d-4d95-82af-d3649027e604"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:38.422627 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.422607 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-xks-ca" pod "a43ccea2-885d-4d95-82af-d3649027e604" (UID: "a43ccea2-885d-4d95-82af-d3649027e604"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:19:38.422780 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.422756 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43ccea2-885d-4d95-82af-d3649027e604-kube-api-access-z64pg" pod "a43ccea2-885d-4d95-82af-d3649027e604" (UID: "a43ccea2-885d-4d95-82af-d3649027e604"). InnerVolumeSpecName "kube-api-access-z64pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:38.521510 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.521434 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:38.521510 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.521506 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ccea2-885d-4d95-82af-d3649027e604-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:38.521721 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:38.521521 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z64pg\" (UniqueName: \"kubernetes.io/projected/a43ccea2-885d-4d95-82af-d3649027e604-kube-api-access-z64pg\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:39.166071 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:39.166040 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" Nov 03 21:19:39.166303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:39.166072 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc" event={"ID":"a43ccea2-885d-4d95-82af-d3649027e604","Type":"ContainerDied","Data":"15e84373ab9ff85d0117c3319aab63efaeb022d0a358071e41d0477f5b34217d"} Nov 03 21:19:39.166303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:39.166114 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e84373ab9ff85d0117c3319aab63efaeb022d0a358071e41d0477f5b34217d" Nov 03 21:19:40.702459 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:40.702413 4711 scope.go:122] "RemoveContainer" containerID="27af624eefb64092683639887d5a984798704ad9638f75176d258eeaf5a03498" Nov 03 21:19:40.712418 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:40.712397 4711 scope.go:122] "RemoveContainer" containerID="540978b2c075440f5c2210ea1256ae3292815fb9823bb43992481b2cbc97786c" Nov 03 21:19:45.680338 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.680295 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz"] Nov 03 21:19:45.680842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.680827 4711 cpu_manager.go:414] "containerMap: removing container" podUID="a43ccea2-885d-4d95-82af-d3649027e604" containerName="revoke-keys" Nov 03 21:19:45.680892 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.680843 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43ccea2-885d-4d95-82af-d3649027e604" containerName="revoke-keys" Nov 03 21:19:45.680932 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.680919 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43ccea2-885d-4d95-82af-d3649027e604" containerName="revoke-keys" Nov 03 21:19:45.686132 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.686110 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:45.692481 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.690990 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:19:45.692651 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.691005 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:19:45.692651 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.691061 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:19:45.692651 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.691247 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:19:45.693302 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.693275 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz"] Nov 03 21:19:45.793382 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.793348 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2vxz\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:45.793608 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.793530 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2vxz\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:45.793608 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.793568 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52vg\" (UniqueName: \"kubernetes.io/projected/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-kube-api-access-r52vg\") pod \"maas-api-revoke-sub-5a29efb3-h2vxz\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:45.894920 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.894882 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2vxz\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:45.894920 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.894921 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2vxz\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:45.895189 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.894943 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r52vg\" (UniqueName: \"kubernetes.io/projected/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-kube-api-access-r52vg\") pod \"maas-api-revoke-sub-5a29efb3-h2vxz\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:45.895576 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.895554 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2vxz\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:45.897332 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.897308 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-h2vxz\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:45.907175 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:45.907151 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52vg\" (UniqueName: \"kubernetes.io/projected/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-kube-api-access-r52vg\") pod \"maas-api-revoke-sub-5a29efb3-h2vxz\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:46.003022 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:46.002931 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:46.279137 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:46.279111 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz"] Nov 03 21:19:46.281088 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:19:46.281054 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f4a1f9_d15d_4b20_8d5a_af3cc10160d6.slice/crio-4cce93d2df9fac1787e6e85f510100dd80b47f887d9c92bdbdff1695ceabf992 WatchSource:0}: Error finding container 4cce93d2df9fac1787e6e85f510100dd80b47f887d9c92bdbdff1695ceabf992: Status 404 returned error can't find the container with id 4cce93d2df9fac1787e6e85f510100dd80b47f887d9c92bdbdff1695ceabf992 Nov 03 21:19:47.211830 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:47.211798 4711 generic.go:356] "Generic (PLEG): container finished" podID="f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6" containerID="b1d8658395c2606870ff7e1c8da5b8fc17b607ac7978ec61d547ff89d8fb8b9d" exitCode=0 Nov 03 21:19:47.212251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:47.211899 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" event={"ID":"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6","Type":"ContainerDied","Data":"b1d8658395c2606870ff7e1c8da5b8fc17b607ac7978ec61d547ff89d8fb8b9d"} Nov 03 21:19:47.212251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:47.211966 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" event={"ID":"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6","Type":"ContainerStarted","Data":"4cce93d2df9fac1787e6e85f510100dd80b47f887d9c92bdbdff1695ceabf992"} Nov 03 21:19:47.543805 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:19:47.543762 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:19:48.635804 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.635723 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:48.756347 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.756314 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-kube-api-access-r52vg\" (UniqueName: \"kubernetes.io/projected/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-kube-api-access-r52vg\") pod \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " Nov 03 21:19:48.756552 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.756471 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-xks-ca\") pod \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " Nov 03 21:19:48.756552 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.756498 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-openshift-ca\") pod \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\" (UID: \"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6\") " Nov 03 21:19:48.756997 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.756970 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-openshift-ca" pod "f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6" (UID: "f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:48.758832 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.758799 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-kube-api-access-r52vg" pod "f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6" (UID: "f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6"). InnerVolumeSpecName "kube-api-access-r52vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:48.758933 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.758879 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-xks-ca" pod "f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6" (UID: "f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:19:48.858069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.857970 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r52vg\" (UniqueName: \"kubernetes.io/projected/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-kube-api-access-r52vg\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:48.858069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.858006 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:48.858069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:48.858023 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:49.249424 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:49.249335 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" Nov 03 21:19:49.249424 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:49.249360 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz" event={"ID":"f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6","Type":"ContainerDied","Data":"4cce93d2df9fac1787e6e85f510100dd80b47f887d9c92bdbdff1695ceabf992"} Nov 03 21:19:49.249695 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:19:49.249431 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cce93d2df9fac1787e6e85f510100dd80b47f887d9c92bdbdff1695ceabf992" Nov 03 21:20:00.751643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:00.751604 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:01.268995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.268950 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l"] Nov 03 21:20:01.271334 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.271301 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjh7l"] Nov 03 21:20:01.294333 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.294293 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n"] Nov 03 21:20:01.294868 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.294849 4711 cpu_manager.go:414] "containerMap: removing container" podUID="f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6" containerName="revoke-keys" Nov 03 21:20:01.294928 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.294869 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6" containerName="revoke-keys" Nov 03 21:20:01.294964 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.294945 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6" containerName="revoke-keys" Nov 03 21:20:01.298264 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.298238 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.307772 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.307745 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:20:01.307943 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.307812 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:20:01.308017 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.307946 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:20:01.308017 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.308011 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:20:01.310258 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.310229 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n"] Nov 03 21:20:01.384943 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.384900 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jff9n\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.385128 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.384988 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nznd8\" (UniqueName: \"kubernetes.io/projected/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-kube-api-access-nznd8\") pod \"maas-api-revoke-sub-3d4156f1-jff9n\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.385128 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.385070 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jff9n\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.486332 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.486292 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jff9n\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.486623 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.486438 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jff9n\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.486623 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.486504 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nznd8\" (UniqueName: \"kubernetes.io/projected/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-kube-api-access-nznd8\") pod \"maas-api-revoke-sub-3d4156f1-jff9n\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.487123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.487099 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jff9n\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.488824 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.488797 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jff9n\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.499147 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.499123 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nznd8\" (UniqueName: \"kubernetes.io/projected/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-kube-api-access-nznd8\") pod \"maas-api-revoke-sub-3d4156f1-jff9n\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.617604 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.617514 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:01.895147 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:01.895116 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n"] Nov 03 21:20:01.897110 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:20:01.897080 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ff9f38_ff1e_4fde_8049_ddc7c5499ec1.slice/crio-957fa930006a542ebe5d77bfec3243b8fb2c5102ecc1d803bff071369ec83f02 WatchSource:0}: Error finding container 957fa930006a542ebe5d77bfec3243b8fb2c5102ecc1d803bff071369ec83f02: Status 404 returned error can't find the container with id 957fa930006a542ebe5d77bfec3243b8fb2c5102ecc1d803bff071369ec83f02 Nov 03 21:20:02.112745 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:02.112709 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:02.323875 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:02.323834 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" event={"ID":"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1","Type":"ContainerStarted","Data":"fe79f3d04440182b4e20f788d3eac571eedeff3fc59a5ee7afdfb0f806718683"} Nov 03 21:20:02.324069 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:02.323898 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" event={"ID":"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1","Type":"ContainerStarted","Data":"957fa930006a542ebe5d77bfec3243b8fb2c5102ecc1d803bff071369ec83f02"} Nov 03 21:20:02.345105 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:02.345049 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" podStartSLOduration=1.345034349 podStartE2EDuration="1.345034349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:02.342743253 +0000 UTC m=+4705.792849305" watchObservedRunningTime="2026-11-03 21:20:02.345034349 +0000 UTC m=+4705.795140390" Nov 03 21:20:02.678390 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:02.678359 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8afb598b-e19d-4f19-82ee-b55761f49a88" path="/var/lib/kubelet/pods/8afb598b-e19d-4f19-82ee-b55761f49a88/volumes" Nov 03 21:20:03.330828 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:03.330795 4711 generic.go:356] "Generic (PLEG): container finished" podID="b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1" containerID="fe79f3d04440182b4e20f788d3eac571eedeff3fc59a5ee7afdfb0f806718683" exitCode=0 Nov 03 21:20:03.331234 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:03.330856 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" event={"ID":"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1","Type":"ContainerDied","Data":"fe79f3d04440182b4e20f788d3eac571eedeff3fc59a5ee7afdfb0f806718683"} Nov 03 21:20:04.473899 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.473866 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:04.519880 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.519831 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-kube-api-access-nznd8\" (UniqueName: \"kubernetes.io/projected/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-kube-api-access-nznd8\") pod \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " Nov 03 21:20:04.520102 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.519926 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-openshift-ca\") pod \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " Nov 03 21:20:04.520102 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.519953 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-xks-ca\") pod \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\" (UID: \"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1\") " Nov 03 21:20:04.520346 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.520312 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-openshift-ca" pod "b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1" (UID: "b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:04.520459 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.520429 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:04.522132 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.522103 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-xks-ca" pod "b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1" (UID: "b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:20:04.522340 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.522320 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-kube-api-access-nznd8" pod "b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1" (UID: "b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1"). InnerVolumeSpecName "kube-api-access-nznd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:04.621880 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.621776 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nznd8\" (UniqueName: \"kubernetes.io/projected/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-kube-api-access-nznd8\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:04.621880 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:04.621816 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:05.342986 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:05.342950 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" Nov 03 21:20:05.343182 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:05.342984 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n" event={"ID":"b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1","Type":"ContainerDied","Data":"957fa930006a542ebe5d77bfec3243b8fb2c5102ecc1d803bff071369ec83f02"} Nov 03 21:20:05.343182 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:05.343038 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="957fa930006a542ebe5d77bfec3243b8fb2c5102ecc1d803bff071369ec83f02" Nov 03 21:20:16.392257 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:16.392220 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:16.829289 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:16.829250 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:16.962637 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:16.962593 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:17.531866 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:20:17.531820 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:20:20.303649 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:20.303605 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:20.440974 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:20.440933 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:21.495942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:21.495894 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:21.639941 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:21.639903 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:21.790844 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:21.790803 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:22.315365 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:22.315307 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:22.997920 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:22.997883 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:23.799623 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:23.799581 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:24.653376 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:24.653340 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:25.496922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:25.496882 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:26.233176 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:26.233136 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:27.011523 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:27.011485 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:27.775738 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:27.775706 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:28.532158 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:28.532119 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:29.324884 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:29.324847 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:30.044735 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:30.044702 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:30.829944 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:30.829906 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:31.655383 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:31.655349 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:32.488138 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:32.488100 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:33.199147 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:33.199104 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:33.895722 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:33.895678 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:34.609406 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:34.609365 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:35.297957 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:35.297915 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:35.430152 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:35.430118 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:35.922560 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:35.922524 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:36.054105 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:36.054068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:36.182551 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:36.182440 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:37.033309 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:37.033274 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:37.823744 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:37.823707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:38.632600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:38.632560 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:38.765229 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:38.765191 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:39.563420 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:39.563380 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:40.343509 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:40.343427 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:40.745004 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:40.744924 4711 scope.go:122] "RemoveContainer" containerID="64085e155fc5c18dbc5ba767e61a82150de2f45e76539c86a0a55c1b6f22d493" Nov 03 21:20:41.244256 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:41.244220 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:42.011269 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:42.011228 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:42.759951 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:42.759912 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:43.423107 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:43.423072 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:44.110847 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:44.110813 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:44.810327 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:44.810296 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:45.506462 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:45.506408 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:46.204597 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:46.204560 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:46.908640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:46.908601 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:47.532593 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:20:47.532555 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:20:47.607500 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:47.607440 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:48.330942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:48.330878 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:49.000928 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:49.000888 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:49.717699 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:49.717663 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:50.517741 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:50.517705 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:51.216989 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:51.216940 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:51.914164 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:51.914132 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:52.623683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:52.623643 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:53.417609 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:53.417563 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:54.113026 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:54.112990 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:54.891860 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:54.891819 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:55.615627 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:55.615576 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:56.312377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:56.312340 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:57.026135 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:57.026100 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:57.706126 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:57.706089 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:58.413628 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:58.413595 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:59.106408 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:59.106378 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:20:59.907463 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:20:59.907419 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:00.599936 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:00.599804 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:00.799396 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:00.799362 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:01.308303 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:01.308253 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:02.007198 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:02.007156 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:02.185108 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:02.185070 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:02.703877 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:02.703831 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:03.400203 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:03.400163 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:04.112470 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:04.112411 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:04.811425 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:04.811379 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:05.513932 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:05.513868 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:06.211805 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:06.211766 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:07.001744 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:07.001705 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:07.698653 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:07.698615 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:08.401791 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:08.401752 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:09.108190 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.108150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:09.435627 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.435550 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9"] Nov 03 21:21:09.436003 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.435989 4711 cpu_manager.go:414] "containerMap: removing container" podUID="b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1" containerName="revoke-keys" Nov 03 21:21:09.436057 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.436004 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1" containerName="revoke-keys" Nov 03 21:21:09.436095 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.436074 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1" containerName="revoke-keys" Nov 03 21:21:09.439349 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.439326 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.447540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.446524 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:21:09.447540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.446533 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:21:09.447540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.446570 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:21:09.447540 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.446398 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:21:09.448711 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.448533 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9"] Nov 03 21:21:09.496088 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.496050 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4097800-wwxh9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.496260 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.496128 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2drpg\" (UniqueName: \"kubernetes.io/projected/864620b6-e214-49f5-908c-106c3b03bab9-kube-api-access-2drpg\") pod \"maas-api-revoke-sub-a4097800-wwxh9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.496260 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.496237 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4097800-wwxh9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.597491 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.597434 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4097800-wwxh9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.597659 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.597538 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2drpg\" (UniqueName: \"kubernetes.io/projected/864620b6-e214-49f5-908c-106c3b03bab9-kube-api-access-2drpg\") pod \"maas-api-revoke-sub-a4097800-wwxh9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.597659 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.597589 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4097800-wwxh9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.598194 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.598173 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4097800-wwxh9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.599761 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.599741 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4097800-wwxh9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.609876 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.609854 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2drpg\" (UniqueName: \"kubernetes.io/projected/864620b6-e214-49f5-908c-106c3b03bab9-kube-api-access-2drpg\") pod \"maas-api-revoke-sub-a4097800-wwxh9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.753996 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.753913 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:09.799495 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:09.799425 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:10.023732 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:10.023698 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6"] Nov 03 21:21:10.026205 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:10.026180 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-98zg6"] Nov 03 21:21:10.035281 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:10.030758 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9"] Nov 03 21:21:10.038303 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:21:10.038272 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864620b6_e214_49f5_908c_106c3b03bab9.slice/crio-5fbf1f0a9e5ecbf59ee105645c366b0771beb18095b650b60aef2d99d238e0f3 WatchSource:0}: Error finding container 5fbf1f0a9e5ecbf59ee105645c366b0771beb18095b650b60aef2d99d238e0f3: Status 404 returned error can't find the container with id 5fbf1f0a9e5ecbf59ee105645c366b0771beb18095b650b60aef2d99d238e0f3 Nov 03 21:21:10.501252 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:10.501211 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:10.678346 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:10.678269 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76115a5f-4577-4d58-9cd5-7b24d406e2bc" path="/var/lib/kubelet/pods/76115a5f-4577-4d58-9cd5-7b24d406e2bc/volumes" Nov 03 21:21:10.734132 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:10.734094 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" event={"ID":"864620b6-e214-49f5-908c-106c3b03bab9","Type":"ContainerStarted","Data":"ceb078245b07d3ab29e52faff5e6181f5d46ab6b7b7b59754508d71fc79c8817"} Nov 03 21:21:10.734301 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:10.734147 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" event={"ID":"864620b6-e214-49f5-908c-106c3b03bab9","Type":"ContainerStarted","Data":"5fbf1f0a9e5ecbf59ee105645c366b0771beb18095b650b60aef2d99d238e0f3"} Nov 03 21:21:10.754842 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:10.754796 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" podStartSLOduration=1.7547828760000002 podStartE2EDuration="1.754782876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:10.751982518 +0000 UTC m=+4774.202088566" watchObservedRunningTime="2026-11-03 21:21:10.754782876 +0000 UTC m=+4774.204888923" Nov 03 21:21:11.201245 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:11.201211 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:11.740784 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:11.740750 4711 generic.go:356] "Generic (PLEG): container finished" podID="864620b6-e214-49f5-908c-106c3b03bab9" containerID="ceb078245b07d3ab29e52faff5e6181f5d46ab6b7b7b59754508d71fc79c8817" exitCode=0 Nov 03 21:21:11.741211 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:11.740831 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" event={"ID":"864620b6-e214-49f5-908c-106c3b03bab9","Type":"ContainerDied","Data":"ceb078245b07d3ab29e52faff5e6181f5d46ab6b7b7b59754508d71fc79c8817"} Nov 03 21:21:11.906377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:11.906339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:12.600242 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:12.600207 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:12.880403 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:12.880378 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:12.929710 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:12.929680 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-xks-ca\") pod \"864620b6-e214-49f5-908c-106c3b03bab9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " Nov 03 21:21:12.929890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:12.929742 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864620b6-e214-49f5-908c-106c3b03bab9-kube-api-access-2drpg\" (UniqueName: \"kubernetes.io/projected/864620b6-e214-49f5-908c-106c3b03bab9-kube-api-access-2drpg\") pod \"864620b6-e214-49f5-908c-106c3b03bab9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " Nov 03 21:21:12.929890 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:12.929830 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-openshift-ca\") pod \"864620b6-e214-49f5-908c-106c3b03bab9\" (UID: \"864620b6-e214-49f5-908c-106c3b03bab9\") " Nov 03 21:21:12.930265 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:12.930236 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-openshift-ca" pod "864620b6-e214-49f5-908c-106c3b03bab9" (UID: "864620b6-e214-49f5-908c-106c3b03bab9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:21:12.931807 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:12.931780 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-xks-ca" pod "864620b6-e214-49f5-908c-106c3b03bab9" (UID: "864620b6-e214-49f5-908c-106c3b03bab9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:12.931916 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:12.931848 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864620b6-e214-49f5-908c-106c3b03bab9-kube-api-access-2drpg" pod "864620b6-e214-49f5-908c-106c3b03bab9" (UID: "864620b6-e214-49f5-908c-106c3b03bab9"). InnerVolumeSpecName "kube-api-access-2drpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:13.031034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:13.031000 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:13.031034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:13.031028 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2drpg\" (UniqueName: \"kubernetes.io/projected/864620b6-e214-49f5-908c-106c3b03bab9-kube-api-access-2drpg\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:13.031233 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:13.031042 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864620b6-e214-49f5-908c-106c3b03bab9-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:13.302606 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:13.302551 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:13.752145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:13.752069 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" Nov 03 21:21:13.752145 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:13.752105 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9" event={"ID":"864620b6-e214-49f5-908c-106c3b03bab9","Type":"ContainerDied","Data":"5fbf1f0a9e5ecbf59ee105645c366b0771beb18095b650b60aef2d99d238e0f3"} Nov 03 21:21:13.752335 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:13.752153 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fbf1f0a9e5ecbf59ee105645c366b0771beb18095b650b60aef2d99d238e0f3" Nov 03 21:21:14.002104 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:14.002068 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:14.699375 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:14.699342 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:15.452539 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:15.452497 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:16.101861 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:16.101821 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:16.901889 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:16.901847 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:17.531492 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:21:17.531454 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:21:17.601763 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:17.601707 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:18.307682 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:18.307647 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:19.000077 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:19.000041 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:19.702171 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:19.702138 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:21.114696 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:21.114658 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:21.857208 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:21.857156 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:22.685208 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:22.685168 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:23.374481 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:23.374426 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:24.084065 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:24.084022 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:24.899521 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:24.899488 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:25.378958 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:25.378920 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:25.601277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:25.601231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:26.306607 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:26.306569 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:26.642255 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:26.642181 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:26.775184 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:26.775150 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:27.090707 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:27.090660 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:27.220034 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:27.219997 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:27.681533 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:27.681494 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:29.023515 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:29.023484 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq"] Nov 03 21:21:29.026151 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:29.026126 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bnmhq"] Nov 03 21:21:30.678699 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:30.678665 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366a155e-df8c-4ed0-b1e3-f74750776407" path="/var/lib/kubelet/pods/366a155e-df8c-4ed0-b1e3-f74750776407/volumes" Nov 03 21:21:32.754952 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:32.754913 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:32.888745 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:32.888699 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:37.995936 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:37.995907 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:21:37.998848 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:37.998831 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:21:38.037847 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:38.037820 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:21:38.040547 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:38.040529 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:21:47.026458 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:47.026408 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k"] Nov 03 21:21:47.030841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:47.030814 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-x8x9k"] Nov 03 21:21:47.531700 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:21:47.531657 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:21:48.678808 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:48.678766 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda7c002-297d-45fb-8229-c86310e88672" path="/var/lib/kubelet/pods/fda7c002-297d-45fb-8229-c86310e88672/volumes" Nov 03 21:21:57.023697 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:57.023661 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz"] Nov 03 21:21:57.029249 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:57.029224 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-x4gdz"] Nov 03 21:21:57.873562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:57.873519 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:58.012470 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:58.012417 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:58.679013 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:58.678973 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81450211-1c91-4e25-9b2a-1f242ef56dd0" path="/var/lib/kubelet/pods/81450211-1c91-4e25-9b2a-1f242ef56dd0/volumes" Nov 03 21:21:59.140165 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:59.140126 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:59.278537 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:59.278504 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:21:59.992063 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:21:59.992020 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:00.705080 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:00.705046 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:00.871839 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:00.871802 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:01.407404 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:01.407367 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:02.236234 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:02.236194 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:02.372015 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:02.371974 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:03.073869 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:03.073822 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:03.204382 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:03.204349 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:03.915299 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:03.915256 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:04.808463 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:04.808415 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:05.603174 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:05.603139 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:06.309456 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:06.309424 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:07.007086 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.007048 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:07.509715 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.509675 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp"] Nov 03 21:22:07.510166 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.510152 4711 cpu_manager.go:414] "containerMap: removing container" podUID="864620b6-e214-49f5-908c-106c3b03bab9" containerName="revoke-keys" Nov 03 21:22:07.510216 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.510167 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864620b6-e214-49f5-908c-106c3b03bab9" containerName="revoke-keys" Nov 03 21:22:07.510278 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.510269 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="864620b6-e214-49f5-908c-106c3b03bab9" containerName="revoke-keys" Nov 03 21:22:07.513794 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.513771 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.519871 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.519837 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:22:07.522385 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.522362 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:22:07.523140 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.523121 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:22:07.523264 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.523184 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:22:07.523762 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.523735 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp"] Nov 03 21:22:07.682189 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.682150 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.682378 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.682233 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjnw4\" (UniqueName: \"kubernetes.io/projected/ba5e162f-0048-4f57-93a7-ab4a04b6a025-kube-api-access-hjnw4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.682378 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.682301 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.707358 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.707326 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:07.783427 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.783334 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.783427 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.783422 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjnw4\" (UniqueName: \"kubernetes.io/projected/ba5e162f-0048-4f57-93a7-ab4a04b6a025-kube-api-access-hjnw4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.783691 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.783500 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.784087 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.784068 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.785788 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.785768 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.798146 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.798120 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjnw4\" (UniqueName: \"kubernetes.io/projected/ba5e162f-0048-4f57-93a7-ab4a04b6a025-kube-api-access-hjnw4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:07.829493 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:07.829470 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:08.108275 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:08.108246 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp"] Nov 03 21:22:08.109483 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:22:08.109458 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5e162f_0048_4f57_93a7_ab4a04b6a025.slice/crio-689bb290e2e500fb52cf46c8af8833db8b5d0951a18e5b970d588424bcaac558 WatchSource:0}: Error finding container 689bb290e2e500fb52cf46c8af8833db8b5d0951a18e5b970d588424bcaac558: Status 404 returned error can't find the container with id 689bb290e2e500fb52cf46c8af8833db8b5d0951a18e5b970d588424bcaac558 Nov 03 21:22:08.411884 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:08.411850 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:09.055272 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:09.055239 4711 generic.go:356] "Generic (PLEG): container finished" podID="ba5e162f-0048-4f57-93a7-ab4a04b6a025" containerID="b7ddfed32fc037accc3e4bf91af1680f41a75c431134416aa0a6a75750660597" exitCode=0 Nov 03 21:22:09.055747 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:09.055321 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" event={"ID":"ba5e162f-0048-4f57-93a7-ab4a04b6a025","Type":"ContainerDied","Data":"b7ddfed32fc037accc3e4bf91af1680f41a75c431134416aa0a6a75750660597"} Nov 03 21:22:09.055747 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:09.055373 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" event={"ID":"ba5e162f-0048-4f57-93a7-ab4a04b6a025","Type":"ContainerStarted","Data":"689bb290e2e500fb52cf46c8af8833db8b5d0951a18e5b970d588424bcaac558"} Nov 03 21:22:09.112955 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:09.112917 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:09.266211 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:09.266179 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:09.806297 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:09.806255 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:10.195801 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.195774 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:10.310233 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.310195 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-xks-ca\") pod \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " Nov 03 21:22:10.310233 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.310238 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5e162f-0048-4f57-93a7-ab4a04b6a025-kube-api-access-hjnw4\" (UniqueName: \"kubernetes.io/projected/ba5e162f-0048-4f57-93a7-ab4a04b6a025-kube-api-access-hjnw4\") pod \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " Nov 03 21:22:10.310565 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.310281 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-openshift-ca\") pod \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\" (UID: \"ba5e162f-0048-4f57-93a7-ab4a04b6a025\") " Nov 03 21:22:10.310793 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.310738 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-openshift-ca" pod "ba5e162f-0048-4f57-93a7-ab4a04b6a025" (UID: "ba5e162f-0048-4f57-93a7-ab4a04b6a025"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:22:10.310879 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.310849 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:10.312435 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.312404 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-xks-ca" pod "ba5e162f-0048-4f57-93a7-ab4a04b6a025" (UID: "ba5e162f-0048-4f57-93a7-ab4a04b6a025"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:10.312579 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.312436 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5e162f-0048-4f57-93a7-ab4a04b6a025-kube-api-access-hjnw4" pod "ba5e162f-0048-4f57-93a7-ab4a04b6a025" (UID: "ba5e162f-0048-4f57-93a7-ab4a04b6a025"). InnerVolumeSpecName "kube-api-access-hjnw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:10.411420 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.411382 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e162f-0048-4f57-93a7-ab4a04b6a025-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:10.411420 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.411416 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjnw4\" (UniqueName: \"kubernetes.io/projected/ba5e162f-0048-4f57-93a7-ab4a04b6a025-kube-api-access-hjnw4\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:10.512953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:10.512920 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:11.067365 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:11.067331 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" Nov 03 21:22:11.067365 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:11.067358 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp" event={"ID":"ba5e162f-0048-4f57-93a7-ab4a04b6a025","Type":"ContainerDied","Data":"689bb290e2e500fb52cf46c8af8833db8b5d0951a18e5b970d588424bcaac558"} Nov 03 21:22:11.067661 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:11.067406 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="689bb290e2e500fb52cf46c8af8833db8b5d0951a18e5b970d588424bcaac558" Nov 03 21:22:11.207838 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:11.207794 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:11.915057 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:11.915021 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:12.613108 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:12.613076 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:13.310614 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:13.310574 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:14.008071 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:14.008037 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:14.712815 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:14.712776 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:15.404998 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:15.404955 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:16.115381 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:16.115339 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:16.848601 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:16.848557 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:17.532423 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:22:17.532378 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:22:17.703770 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:17.703731 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:17.838926 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:17.838811 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:18.218472 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:18.218340 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:18.371311 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:18.371274 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:18.478000 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:18.477909 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp"] Nov 03 21:22:18.480388 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:18.480362 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d9b592-h9vtp"] Nov 03 21:22:18.504064 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:18.504032 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:18.678454 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:18.678405 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5e162f-0048-4f57-93a7-ab4a04b6a025" path="/var/lib/kubelet/pods/ba5e162f-0048-4f57-93a7-ab4a04b6a025/volumes" Nov 03 21:22:18.917509 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:18.917466 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:22:20.084910 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.084870 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n"] Nov 03 21:22:20.089744 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.089714 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jff9n"] Nov 03 21:22:20.107683 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.107649 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76"] Nov 03 21:22:20.108210 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.108189 4711 cpu_manager.go:414] "containerMap: removing container" podUID="ba5e162f-0048-4f57-93a7-ab4a04b6a025" containerName="revoke-keys" Nov 03 21:22:20.108210 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.108209 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5e162f-0048-4f57-93a7-ab4a04b6a025" containerName="revoke-keys" Nov 03 21:22:20.108383 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.108317 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5e162f-0048-4f57-93a7-ab4a04b6a025" containerName="revoke-keys" Nov 03 21:22:20.111488 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.111467 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.117716 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.117691 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:22:20.117935 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.117699 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:22:20.117935 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.117874 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:22:20.118695 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.118675 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:22:20.122409 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.122387 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76"] Nov 03 21:22:20.208775 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.208731 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s6c76\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.208962 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.208784 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms66j\" (UniqueName: \"kubernetes.io/projected/16acce55-4861-46fc-84a3-6035346e9cf9-kube-api-access-ms66j\") pod \"maas-api-revoke-sub-3d4156f1-s6c76\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.208962 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.208874 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s6c76\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.309793 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.309752 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s6c76\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.310019 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.309804 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms66j\" (UniqueName: \"kubernetes.io/projected/16acce55-4861-46fc-84a3-6035346e9cf9-kube-api-access-ms66j\") pod \"maas-api-revoke-sub-3d4156f1-s6c76\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.310019 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.309833 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s6c76\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.310475 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.310428 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s6c76\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.312384 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.312360 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s6c76\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.322694 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.322666 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms66j\" (UniqueName: \"kubernetes.io/projected/16acce55-4861-46fc-84a3-6035346e9cf9-kube-api-access-ms66j\") pod \"maas-api-revoke-sub-3d4156f1-s6c76\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.426927 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.426847 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:20.678809 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.678726 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1" path="/var/lib/kubelet/pods/b7ff9f38-ff1e-4fde-8049-ddc7c5499ec1/volumes" Nov 03 21:22:20.702548 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:20.702525 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76"] Nov 03 21:22:20.704639 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:22:20.704605 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16acce55_4861_46fc_84a3_6035346e9cf9.slice/crio-da337a88e5cd8750c9a2582c93cec4c703c755bd41d6718d828cc2c519af8df9 WatchSource:0}: Error finding container da337a88e5cd8750c9a2582c93cec4c703c755bd41d6718d828cc2c519af8df9: Status 404 returned error can't find the container with id da337a88e5cd8750c9a2582c93cec4c703c755bd41d6718d828cc2c519af8df9 Nov 03 21:22:21.129197 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:21.129161 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" event={"ID":"16acce55-4861-46fc-84a3-6035346e9cf9","Type":"ContainerStarted","Data":"7fc49fbe9d99158b86a3232f39bf10c1d7f9a02acfcfceafca12bfa52e3f698b"} Nov 03 21:22:21.129693 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:21.129218 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" event={"ID":"16acce55-4861-46fc-84a3-6035346e9cf9","Type":"ContainerStarted","Data":"da337a88e5cd8750c9a2582c93cec4c703c755bd41d6718d828cc2c519af8df9"} Nov 03 21:22:21.152843 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:21.152789 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" podStartSLOduration=1.152775481 podStartE2EDuration="1.152775481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:21.15062401 +0000 UTC m=+4844.600730049" watchObservedRunningTime="2026-11-03 21:22:21.152775481 +0000 UTC m=+4844.602881528" Nov 03 21:22:22.136438 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:22.136402 4711 generic.go:356] "Generic (PLEG): container finished" podID="16acce55-4861-46fc-84a3-6035346e9cf9" containerID="7fc49fbe9d99158b86a3232f39bf10c1d7f9a02acfcfceafca12bfa52e3f698b" exitCode=0 Nov 03 21:22:22.136942 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:22.136477 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" event={"ID":"16acce55-4861-46fc-84a3-6035346e9cf9","Type":"ContainerDied","Data":"7fc49fbe9d99158b86a3232f39bf10c1d7f9a02acfcfceafca12bfa52e3f698b"} Nov 03 21:22:23.051922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.051887 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw"] Nov 03 21:22:23.056183 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.056148 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-fsnjw"] Nov 03 21:22:23.284071 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.284044 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:23.342045 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.341939 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-openshift-ca\") pod \"16acce55-4861-46fc-84a3-6035346e9cf9\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " Nov 03 21:22:23.342045 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.342016 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-xks-ca\") pod \"16acce55-4861-46fc-84a3-6035346e9cf9\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " Nov 03 21:22:23.342297 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.342191 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16acce55-4861-46fc-84a3-6035346e9cf9-kube-api-access-ms66j\" (UniqueName: \"kubernetes.io/projected/16acce55-4861-46fc-84a3-6035346e9cf9-kube-api-access-ms66j\") pod \"16acce55-4861-46fc-84a3-6035346e9cf9\" (UID: \"16acce55-4861-46fc-84a3-6035346e9cf9\") " Nov 03 21:22:23.342359 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.342315 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-openshift-ca" pod "16acce55-4861-46fc-84a3-6035346e9cf9" (UID: "16acce55-4861-46fc-84a3-6035346e9cf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:22:23.342572 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.342556 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:23.344175 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.344151 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16acce55-4861-46fc-84a3-6035346e9cf9-kube-api-access-ms66j" pod "16acce55-4861-46fc-84a3-6035346e9cf9" (UID: "16acce55-4861-46fc-84a3-6035346e9cf9"). InnerVolumeSpecName "kube-api-access-ms66j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:23.344419 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.344401 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-xks-ca" pod "16acce55-4861-46fc-84a3-6035346e9cf9" (UID: "16acce55-4861-46fc-84a3-6035346e9cf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:23.443829 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.443788 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16acce55-4861-46fc-84a3-6035346e9cf9-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:23.443829 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:23.443824 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms66j\" (UniqueName: \"kubernetes.io/projected/16acce55-4861-46fc-84a3-6035346e9cf9-kube-api-access-ms66j\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:24.149536 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:24.149508 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" Nov 03 21:22:24.149745 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:24.149536 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76" event={"ID":"16acce55-4861-46fc-84a3-6035346e9cf9","Type":"ContainerDied","Data":"da337a88e5cd8750c9a2582c93cec4c703c755bd41d6718d828cc2c519af8df9"} Nov 03 21:22:24.149745 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:24.149583 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da337a88e5cd8750c9a2582c93cec4c703c755bd41d6718d828cc2c519af8df9" Nov 03 21:22:24.678766 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:24.678725 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38eaf7ed-d304-4049-8d09-946fb484b688" path="/var/lib/kubelet/pods/38eaf7ed-d304-4049-8d09-946fb484b688/volumes" Nov 03 21:22:34.023549 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:34.023492 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq"] Nov 03 21:22:34.026392 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:34.026357 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rbxvq"] Nov 03 21:22:34.679185 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:34.679153 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5c757f-b54b-4d46-9208-9341546e9aab" path="/var/lib/kubelet/pods/5d5c757f-b54b-4d46-9208-9341546e9aab/volumes" Nov 03 21:22:40.776257 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:40.776220 4711 scope.go:122] "RemoveContainer" containerID="c39985f5fd99cb24a33ee5eacff9f26e93a435314e8f0e6c4b7804cbb5e7cdfe" Nov 03 21:22:40.785612 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:40.785588 4711 scope.go:122] "RemoveContainer" containerID="bfda6f2b526b315d30661dc5e047c31be2c27ceac7407dbec9b73d5c8622a940" Nov 03 21:22:47.531621 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:22:47.531540 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:22:54.033215 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:54.033175 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t"] Nov 03 21:22:54.035764 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:54.035735 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-slf8t"] Nov 03 21:22:54.678515 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:22:54.678482 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ece04e5-844c-43d5-91e2-00765459c610" path="/var/lib/kubelet/pods/6ece04e5-844c-43d5-91e2-00765459c610/volumes" Nov 03 21:23:00.930418 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:00.930358 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:02.273338 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:02.273296 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:23:11.022951 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:11.022912 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7"] Nov 03 21:23:11.025748 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:11.025717 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-kkqf7"] Nov 03 21:23:12.679298 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:12.679261 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3b609b-31c4-46a1-a7c2-72b1ce7c7721" path="/var/lib/kubelet/pods/0a3b609b-31c4-46a1-a7c2-72b1ce7c7721/volumes" Nov 03 21:23:17.530601 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:23:17.530563 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:23:27.023364 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:27.023324 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr"] Nov 03 21:23:27.026255 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:27.026224 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rw7qr"] Nov 03 21:23:28.678735 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:28.678700 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aaf955d-a69b-4136-bee0-ede342c65b8c" path="/var/lib/kubelet/pods/2aaf955d-a69b-4136-bee0-ede342c65b8c/volumes" Nov 03 21:23:40.812545 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:40.812506 4711 scope.go:122] "RemoveContainer" containerID="aafc4d9b07fac03b8d66a4125c1301dcb5ee2da34a69ecca56e9b3a3d78794fb" Nov 03 21:23:40.821247 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:40.821228 4711 scope.go:122] "RemoveContainer" containerID="e02e14d105081d6239b117cc75286658d162412ec0f677b6ab2847f15b342745" Nov 03 21:23:40.830933 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:40.830913 4711 scope.go:122] "RemoveContainer" containerID="d4e0eef1cd1c841a2025ef7785b0f3d042c7c57a969a1b7b9db8eee849196127" Nov 03 21:23:40.839313 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:40.839297 4711 scope.go:122] "RemoveContainer" containerID="9453ec923f095ebe54918f1c0f050c956cfb6b51fcd24e29e6a9ba681f108a76" Nov 03 21:23:41.025541 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:41.025486 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z"] Nov 03 21:23:41.027995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:41.027971 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sp28z"] Nov 03 21:23:42.678600 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:23:42.678565 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a375b334-ec91-4661-9bf6-03139a60920a" path="/var/lib/kubelet/pods/a375b334-ec91-4661-9bf6-03139a60920a/volumes" Nov 03 21:23:47.530770 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:23:47.530731 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:24:00.976882 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:00.976843 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:02.329951 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:02.329915 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:07.742716 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.742635 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt"] Nov 03 21:24:07.743273 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.743252 4711 cpu_manager.go:414] "containerMap: removing container" podUID="16acce55-4861-46fc-84a3-6035346e9cf9" containerName="revoke-keys" Nov 03 21:24:07.743362 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.743275 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16acce55-4861-46fc-84a3-6035346e9cf9" containerName="revoke-keys" Nov 03 21:24:07.743418 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.743379 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="16acce55-4861-46fc-84a3-6035346e9cf9" containerName="revoke-keys" Nov 03 21:24:07.746647 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.746624 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:07.754238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.754211 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:24:07.754474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.754297 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:24:07.754474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.754388 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:24:07.755542 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.754523 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:24:07.757518 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.757491 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt"] Nov 03 21:24:07.903530 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.903495 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshnf\" (UniqueName: \"kubernetes.io/projected/6588d280-e10c-4457-8fd6-50c2385d809e-kube-api-access-rshnf\") pod \"maas-api-revoke-sub-d576e68b-gdgjt\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:07.903704 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.903578 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gdgjt\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:07.903757 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:07.903722 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gdgjt\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:08.005043 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.005010 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gdgjt\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:08.005243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.005095 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gdgjt\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:08.005243 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.005153 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshnf\" (UniqueName: \"kubernetes.io/projected/6588d280-e10c-4457-8fd6-50c2385d809e-kube-api-access-rshnf\") pod \"maas-api-revoke-sub-d576e68b-gdgjt\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:08.005858 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.005832 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gdgjt\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:08.007377 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.007358 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gdgjt\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:08.017743 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.017721 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshnf\" (UniqueName: \"kubernetes.io/projected/6588d280-e10c-4457-8fd6-50c2385d809e-kube-api-access-rshnf\") pod \"maas-api-revoke-sub-d576e68b-gdgjt\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:08.064013 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.063986 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:08.341147 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.341121 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt"] Nov 03 21:24:08.343273 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:24:08.343243 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6588d280_e10c_4457_8fd6_50c2385d809e.slice/crio-8d54d1ff559662b2f3e0036dd5e974e568858bb3da13bb4ef1becb68017fbc84 WatchSource:0}: Error finding container 8d54d1ff559662b2f3e0036dd5e974e568858bb3da13bb4ef1becb68017fbc84: Status 404 returned error can't find the container with id 8d54d1ff559662b2f3e0036dd5e974e568858bb3da13bb4ef1becb68017fbc84 Nov 03 21:24:08.750061 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.750021 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" event={"ID":"6588d280-e10c-4457-8fd6-50c2385d809e","Type":"ContainerStarted","Data":"b145d2c989562989e21d9b0f41ac3a336003fe2dbda881251cc7ec080eefc9f8"} Nov 03 21:24:08.750528 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.750089 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" event={"ID":"6588d280-e10c-4457-8fd6-50c2385d809e","Type":"ContainerStarted","Data":"8d54d1ff559662b2f3e0036dd5e974e568858bb3da13bb4ef1becb68017fbc84"} Nov 03 21:24:08.770035 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:08.769989 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" podStartSLOduration=1.7699747399999999 podStartE2EDuration="1.76997474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:08.768708454 +0000 UTC m=+4952.218814502" watchObservedRunningTime="2026-11-03 21:24:08.76997474 +0000 UTC m=+4952.220080785" Nov 03 21:24:09.757646 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:09.757613 4711 generic.go:356] "Generic (PLEG): container finished" podID="6588d280-e10c-4457-8fd6-50c2385d809e" containerID="b145d2c989562989e21d9b0f41ac3a336003fe2dbda881251cc7ec080eefc9f8" exitCode=0 Nov 03 21:24:09.758066 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:09.757699 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" event={"ID":"6588d280-e10c-4457-8fd6-50c2385d809e","Type":"ContainerDied","Data":"b145d2c989562989e21d9b0f41ac3a336003fe2dbda881251cc7ec080eefc9f8"} Nov 03 21:24:10.896138 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:10.896110 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:11.035098 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.035003 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-openshift-ca\") pod \"6588d280-e10c-4457-8fd6-50c2385d809e\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " Nov 03 21:24:11.035098 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.035082 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6588d280-e10c-4457-8fd6-50c2385d809e-kube-api-access-rshnf\" (UniqueName: \"kubernetes.io/projected/6588d280-e10c-4457-8fd6-50c2385d809e-kube-api-access-rshnf\") pod \"6588d280-e10c-4457-8fd6-50c2385d809e\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " Nov 03 21:24:11.035400 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.035134 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-xks-ca\") pod \"6588d280-e10c-4457-8fd6-50c2385d809e\" (UID: \"6588d280-e10c-4457-8fd6-50c2385d809e\") " Nov 03 21:24:11.035400 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.035386 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-openshift-ca" pod "6588d280-e10c-4457-8fd6-50c2385d809e" (UID: "6588d280-e10c-4457-8fd6-50c2385d809e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:11.037207 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.037183 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-xks-ca" pod "6588d280-e10c-4457-8fd6-50c2385d809e" (UID: "6588d280-e10c-4457-8fd6-50c2385d809e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:11.037323 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.037257 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6588d280-e10c-4457-8fd6-50c2385d809e-kube-api-access-rshnf" pod "6588d280-e10c-4457-8fd6-50c2385d809e" (UID: "6588d280-e10c-4457-8fd6-50c2385d809e"). InnerVolumeSpecName "kube-api-access-rshnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:11.135889 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.135858 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:11.135889 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.135883 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshnf\" (UniqueName: \"kubernetes.io/projected/6588d280-e10c-4457-8fd6-50c2385d809e-kube-api-access-rshnf\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:11.135889 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.135893 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6588d280-e10c-4457-8fd6-50c2385d809e-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:11.769217 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.769190 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" Nov 03 21:24:11.769393 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.769185 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt" event={"ID":"6588d280-e10c-4457-8fd6-50c2385d809e","Type":"ContainerDied","Data":"8d54d1ff559662b2f3e0036dd5e974e568858bb3da13bb4ef1becb68017fbc84"} Nov 03 21:24:11.769393 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:11.769312 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d54d1ff559662b2f3e0036dd5e974e568858bb3da13bb4ef1becb68017fbc84" Nov 03 21:24:17.533210 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:24:17.533104 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:24:23.806643 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:23.806601 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:24:23.807358 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:23.807322 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:24:23.818582 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:23.818547 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:24:23.818831 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:23.818803 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:24:25.302295 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.302250 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb"] Nov 03 21:24:25.302945 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.302920 4711 cpu_manager.go:414] "containerMap: removing container" podUID="6588d280-e10c-4457-8fd6-50c2385d809e" containerName="revoke-keys" Nov 03 21:24:25.303003 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.302948 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6588d280-e10c-4457-8fd6-50c2385d809e" containerName="revoke-keys" Nov 03 21:24:25.303137 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.303125 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="6588d280-e10c-4457-8fd6-50c2385d809e" containerName="revoke-keys" Nov 03 21:24:25.306695 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.306672 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.314381 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.313096 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:24:25.314381 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.313093 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:24:25.314381 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.313946 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:24:25.314381 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.314351 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8fcjp\"" Nov 03 21:24:25.317589 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.317558 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb"] Nov 03 21:24:25.407916 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.407876 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-r27lb\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.408087 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.408009 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rpfm\" (UniqueName: \"kubernetes.io/projected/76283c15-c9bc-4d94-a136-e8a36d2a225b-kube-api-access-4rpfm\") pod \"maas-api-revoke-sub-d2bc93bc-r27lb\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.408087 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.408053 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-r27lb\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.509110 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.509052 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rpfm\" (UniqueName: \"kubernetes.io/projected/76283c15-c9bc-4d94-a136-e8a36d2a225b-kube-api-access-4rpfm\") pod \"maas-api-revoke-sub-d2bc93bc-r27lb\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.509110 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.509115 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-r27lb\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.509336 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.509183 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-r27lb\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.509816 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.509794 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-r27lb\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.511606 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.511585 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-r27lb\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.521673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.521647 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rpfm\" (UniqueName: \"kubernetes.io/projected/76283c15-c9bc-4d94-a136-e8a36d2a225b-kube-api-access-4rpfm\") pod \"maas-api-revoke-sub-d2bc93bc-r27lb\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.634894 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.634802 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:25.911238 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:25.911209 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb"] Nov 03 21:24:25.913426 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:24:25.913396 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76283c15_c9bc_4d94_a136_e8a36d2a225b.slice/crio-b135fcee9635aff4224b50442486d6003425d56bf6eba06616f7f30d68bf15e9 WatchSource:0}: Error finding container b135fcee9635aff4224b50442486d6003425d56bf6eba06616f7f30d68bf15e9: Status 404 returned error can't find the container with id b135fcee9635aff4224b50442486d6003425d56bf6eba06616f7f30d68bf15e9 Nov 03 21:24:26.864318 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:26.864284 4711 generic.go:356] "Generic (PLEG): container finished" podID="76283c15-c9bc-4d94-a136-e8a36d2a225b" containerID="fc1ffb15960452978c2f13c7433d64011100047b7fff6f565f321e0b1300d7f7" exitCode=0 Nov 03 21:24:26.864796 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:26.864377 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" event={"ID":"76283c15-c9bc-4d94-a136-e8a36d2a225b","Type":"ContainerDied","Data":"fc1ffb15960452978c2f13c7433d64011100047b7fff6f565f321e0b1300d7f7"} Nov 03 21:24:26.864796 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:26.864431 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" event={"ID":"76283c15-c9bc-4d94-a136-e8a36d2a225b","Type":"ContainerStarted","Data":"b135fcee9635aff4224b50442486d6003425d56bf6eba06616f7f30d68bf15e9"} Nov 03 21:24:28.020619 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.020583 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:28.134364 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.134271 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76283c15-c9bc-4d94-a136-e8a36d2a225b-kube-api-access-4rpfm\" (UniqueName: \"kubernetes.io/projected/76283c15-c9bc-4d94-a136-e8a36d2a225b-kube-api-access-4rpfm\") pod \"76283c15-c9bc-4d94-a136-e8a36d2a225b\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " Nov 03 21:24:28.134364 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.134337 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-openshift-ca\") pod \"76283c15-c9bc-4d94-a136-e8a36d2a225b\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " Nov 03 21:24:28.134602 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.134510 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-xks-ca\") pod \"76283c15-c9bc-4d94-a136-e8a36d2a225b\" (UID: \"76283c15-c9bc-4d94-a136-e8a36d2a225b\") " Nov 03 21:24:28.134765 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.134733 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-openshift-ca" pod "76283c15-c9bc-4d94-a136-e8a36d2a225b" (UID: "76283c15-c9bc-4d94-a136-e8a36d2a225b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:28.134888 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.134836 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-openshift-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:28.136585 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.136562 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76283c15-c9bc-4d94-a136-e8a36d2a225b-kube-api-access-4rpfm" pod "76283c15-c9bc-4d94-a136-e8a36d2a225b" (UID: "76283c15-c9bc-4d94-a136-e8a36d2a225b"). InnerVolumeSpecName "kube-api-access-4rpfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:28.136681 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.136622 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-xks-ca" pod "76283c15-c9bc-4d94-a136-e8a36d2a225b" (UID: "76283c15-c9bc-4d94-a136-e8a36d2a225b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:28.236123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.236082 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76283c15-c9bc-4d94-a136-e8a36d2a225b-maas-api-xks-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:28.236123 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.236124 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rpfm\" (UniqueName: \"kubernetes.io/projected/76283c15-c9bc-4d94-a136-e8a36d2a225b-kube-api-access-4rpfm\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:28.875850 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.875821 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" Nov 03 21:24:28.876037 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.875868 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb" event={"ID":"76283c15-c9bc-4d94-a136-e8a36d2a225b","Type":"ContainerDied","Data":"b135fcee9635aff4224b50442486d6003425d56bf6eba06616f7f30d68bf15e9"} Nov 03 21:24:28.876037 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:28.875929 4711 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b135fcee9635aff4224b50442486d6003425d56bf6eba06616f7f30d68bf15e9" Nov 03 21:24:37.216117 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.216074 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t"] Nov 03 21:24:37.216744 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.216715 4711 cpu_manager.go:414] "containerMap: removing container" podUID="76283c15-c9bc-4d94-a136-e8a36d2a225b" containerName="revoke-keys" Nov 03 21:24:37.216744 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.216745 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76283c15-c9bc-4d94-a136-e8a36d2a225b" containerName="revoke-keys" Nov 03 21:24:37.216945 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.216896 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="76283c15-c9bc-4d94-a136-e8a36d2a225b" containerName="revoke-keys" Nov 03 21:24:37.220470 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.220426 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.231957 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.231925 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 21:24:37.232871 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.232793 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qqpwn\"" Nov 03 21:24:37.233544 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.233520 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 21:24:37.233662 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.233595 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 21:24:37.233732 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.233711 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:24:37.240231 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.240207 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t"] Nov 03 21:24:37.336277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.336238 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382ef016-7441-40f0-8151-0ebab3e5bd30-openshift-service-ca\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.336277 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.336276 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-metrics-certs\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.336562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.336305 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-maas-api-tls\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.336562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.336472 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqpb9\" (UniqueName: \"kubernetes.io/projected/382ef016-7441-40f0-8151-0ebab3e5bd30-kube-api-access-sqpb9\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.437836 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.437803 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqpb9\" (UniqueName: \"kubernetes.io/projected/382ef016-7441-40f0-8151-0ebab3e5bd30-kube-api-access-sqpb9\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.438054 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.437903 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382ef016-7441-40f0-8151-0ebab3e5bd30-openshift-service-ca\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.438054 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.437928 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-metrics-certs\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.438054 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.437984 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-maas-api-tls\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.438684 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.438661 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382ef016-7441-40f0-8151-0ebab3e5bd30-openshift-service-ca\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.440415 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.440392 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-metrics-certs\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.440495 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.440437 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-maas-api-tls\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.450769 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.450747 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqpb9\" (UniqueName: \"kubernetes.io/projected/382ef016-7441-40f0-8151-0ebab3e5bd30-kube-api-access-sqpb9\") pod \"maas-api-6474999cb4-mnp8t\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.540823 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.540790 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:37.819570 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.819537 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t"] Nov 03 21:24:37.820539 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:24:37.820516 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382ef016_7441_40f0_8151_0ebab3e5bd30.slice/crio-11bd1474227862112ae001988cda68491289eed4277af65080c540a1b9273e28 WatchSource:0}: Error finding container 11bd1474227862112ae001988cda68491289eed4277af65080c540a1b9273e28: Status 404 returned error can't find the container with id 11bd1474227862112ae001988cda68491289eed4277af65080c540a1b9273e28 Nov 03 21:24:37.821921 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.821900 4711 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:24:37.924992 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:37.924958 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" event={"ID":"382ef016-7441-40f0-8151-0ebab3e5bd30","Type":"ContainerStarted","Data":"11bd1474227862112ae001988cda68491289eed4277af65080c540a1b9273e28"} Nov 03 21:24:38.931676 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:38.931639 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" event={"ID":"382ef016-7441-40f0-8151-0ebab3e5bd30","Type":"ContainerStarted","Data":"7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66"} Nov 03 21:24:38.932089 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:38.931788 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:38.953767 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:38.953715 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" podStartSLOduration=1.437518077 podStartE2EDuration="1.953700904s" totalImagesPullingTime="516.182827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:38.952577284 +0000 UTC m=+4982.402683331" watchObservedRunningTime="2026-11-03 21:24:38.953700904 +0000 UTC m=+4982.403806952" Nov 03 21:24:39.029656 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:39.024893 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc"] Nov 03 21:24:39.032586 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:39.032557 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-q4ccc"] Nov 03 21:24:40.166270 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:40.166231 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:40.300353 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:40.300317 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:24:40.678575 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:40.678543 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43ccea2-885d-4d95-82af-d3649027e604" path="/var/lib/kubelet/pods/a43ccea2-885d-4d95-82af-d3649027e604/volumes" Nov 03 21:24:40.871098 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:40.871066 4711 scope.go:122] "RemoveContainer" containerID="f7c86d5f4dc4cece0502f147096b7517c30def81f6aedd889db02d91f0b86d4f" Nov 03 21:24:40.879690 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:40.879667 4711 scope.go:122] "RemoveContainer" containerID="753f446e859f6a4e56a4468231fdde50eac536025eece4cecd59ff9b5f362af1" Nov 03 21:24:40.887936 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:40.887919 4711 scope.go:122] "RemoveContainer" containerID="b0d2cac2084f68cb1ff489caa998d3e467612716e36eb2d347257712243ef802" Nov 03 21:24:40.895640 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:40.895618 4711 scope.go:122] "RemoveContainer" containerID="712ae9e3db951b34a0009061dbbcb333bbfdad3a8c191c1d9a63edd9d5e9d741" Nov 03 21:24:44.627629 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.627585 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t"] Nov 03 21:24:44.628117 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.627856 4711 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" podUID="382ef016-7441-40f0-8151-0ebab3e5bd30" containerName="maas-api" containerID="cri-o://7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66" gracePeriod=30 Nov 03 21:24:44.633650 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.633619 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:44.660368 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.660337 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb"] Nov 03 21:24:44.664573 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.664543 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.698126 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.698096 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb"] Nov 03 21:24:44.717940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.717907 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d63895-e621-40e6-8cd3-0c43df29016f-openshift-service-ca\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.718113 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.717950 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d63895-e621-40e6-8cd3-0c43df29016f-metrics-certs\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.718158 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.718134 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fftxk\" (UniqueName: \"kubernetes.io/projected/60d63895-e621-40e6-8cd3-0c43df29016f-kube-api-access-fftxk\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.718192 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.718177 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60d63895-e621-40e6-8cd3-0c43df29016f-maas-api-tls\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.819673 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.819632 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d63895-e621-40e6-8cd3-0c43df29016f-openshift-service-ca\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.819859 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.819711 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d63895-e621-40e6-8cd3-0c43df29016f-metrics-certs\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.819859 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.819848 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fftxk\" (UniqueName: \"kubernetes.io/projected/60d63895-e621-40e6-8cd3-0c43df29016f-kube-api-access-fftxk\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.819988 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.819888 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60d63895-e621-40e6-8cd3-0c43df29016f-maas-api-tls\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.820369 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.820340 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60d63895-e621-40e6-8cd3-0c43df29016f-openshift-service-ca\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.823009 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.822978 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60d63895-e621-40e6-8cd3-0c43df29016f-maas-api-tls\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.823499 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.823428 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d63895-e621-40e6-8cd3-0c43df29016f-metrics-certs\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.832863 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.832830 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fftxk\" (UniqueName: \"kubernetes.io/projected/60d63895-e621-40e6-8cd3-0c43df29016f-kube-api-access-fftxk\") pod \"maas-api-6b4d8b8448-4m7sb\" (UID: \"60d63895-e621-40e6-8cd3-0c43df29016f\") " pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:44.887039 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.886977 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:44.920674 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.920641 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-maas-api-tls\") pod \"382ef016-7441-40f0-8151-0ebab3e5bd30\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " Nov 03 21:24:44.920901 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.920706 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382ef016-7441-40f0-8151-0ebab3e5bd30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382ef016-7441-40f0-8151-0ebab3e5bd30-openshift-service-ca\") pod \"382ef016-7441-40f0-8151-0ebab3e5bd30\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " Nov 03 21:24:44.920901 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.920827 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-metrics-certs\") pod \"382ef016-7441-40f0-8151-0ebab3e5bd30\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " Nov 03 21:24:44.920901 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.920854 4711 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382ef016-7441-40f0-8151-0ebab3e5bd30-kube-api-access-sqpb9\" (UniqueName: \"kubernetes.io/projected/382ef016-7441-40f0-8151-0ebab3e5bd30-kube-api-access-sqpb9\") pod \"382ef016-7441-40f0-8151-0ebab3e5bd30\" (UID: \"382ef016-7441-40f0-8151-0ebab3e5bd30\") " Nov 03 21:24:44.921126 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.921091 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382ef016-7441-40f0-8151-0ebab3e5bd30-openshift-service-ca" pod "382ef016-7441-40f0-8151-0ebab3e5bd30" (UID: "382ef016-7441-40f0-8151-0ebab3e5bd30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:44.922953 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.922926 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-maas-api-tls" pod "382ef016-7441-40f0-8151-0ebab3e5bd30" (UID: "382ef016-7441-40f0-8151-0ebab3e5bd30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:44.923161 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.923129 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382ef016-7441-40f0-8151-0ebab3e5bd30-kube-api-access-sqpb9" pod "382ef016-7441-40f0-8151-0ebab3e5bd30" (UID: "382ef016-7441-40f0-8151-0ebab3e5bd30"). InnerVolumeSpecName "kube-api-access-sqpb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:44.923255 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.923182 4711 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-metrics-certs" pod "382ef016-7441-40f0-8151-0ebab3e5bd30" (UID: "382ef016-7441-40f0-8151-0ebab3e5bd30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:44.964343 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.964308 4711 generic.go:356] "Generic (PLEG): container finished" podID="382ef016-7441-40f0-8151-0ebab3e5bd30" containerID="7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66" exitCode=0 Nov 03 21:24:44.964513 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.964360 4711 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" Nov 03 21:24:44.964513 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.964394 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" event={"ID":"382ef016-7441-40f0-8151-0ebab3e5bd30","Type":"ContainerDied","Data":"7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66"} Nov 03 21:24:44.964513 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.964481 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t" event={"ID":"382ef016-7441-40f0-8151-0ebab3e5bd30","Type":"ContainerDied","Data":"11bd1474227862112ae001988cda68491289eed4277af65080c540a1b9273e28"} Nov 03 21:24:44.964513 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.964510 4711 scope.go:122] "RemoveContainer" containerID="7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66" Nov 03 21:24:44.974679 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.974662 4711 scope.go:122] "RemoveContainer" containerID="7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66" Nov 03 21:24:44.974942 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:24:44.974921 4711 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66\": container with ID starting with 7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66 not found: ID does not exist" containerID="7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66" Nov 03 21:24:44.975050 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.974948 4711 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66"} err="failed to get container status \"7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66\": rpc error: code = NotFound desc = could not find container \"7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66\": container with ID starting with 7560c8a0b59b0f0be388f8b1cda0976e57956f847c0f267d4cbabe9d79a7de66 not found: ID does not exist" Nov 03 21:24:44.992292 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.992266 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t"] Nov 03 21:24:44.994312 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:44.994290 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6474999cb4-mnp8t"] Nov 03 21:24:45.007869 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.007843 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:45.021995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.021949 4711 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-maas-api-tls\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:45.021995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.021984 4711 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382ef016-7441-40f0-8151-0ebab3e5bd30-openshift-service-ca\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:45.021995 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.021995 4711 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/382ef016-7441-40f0-8151-0ebab3e5bd30-metrics-certs\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:45.022132 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.022003 4711 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqpb9\" (UniqueName: \"kubernetes.io/projected/382ef016-7441-40f0-8151-0ebab3e5bd30-kube-api-access-sqpb9\") on node \"ip-10-0-142-5.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:45.290273 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.290254 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb"] Nov 03 21:24:45.292783 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:24:45.292732 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d63895_e621_40e6_8cd3_0c43df29016f.slice/crio-9fca2c91530d50279e0e26a11932fa9e8072f3d82b679ce9b8fb32be53160628 WatchSource:0}: Error finding container 9fca2c91530d50279e0e26a11932fa9e8072f3d82b679ce9b8fb32be53160628: Status 404 returned error can't find the container with id 9fca2c91530d50279e0e26a11932fa9e8072f3d82b679ce9b8fb32be53160628 Nov 03 21:24:45.971492 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.971382 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" event={"ID":"60d63895-e621-40e6-8cd3-0c43df29016f","Type":"ContainerStarted","Data":"d42ac47ccedbee45fe0e3f421fe089c1e83f1d5da6bfed06513f1d2872f3c8fe"} Nov 03 21:24:45.971492 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.971468 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:24:45.971959 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.971502 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" event={"ID":"60d63895-e621-40e6-8cd3-0c43df29016f","Type":"ContainerStarted","Data":"9fca2c91530d50279e0e26a11932fa9e8072f3d82b679ce9b8fb32be53160628"} Nov 03 21:24:45.992659 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:45.992607 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" podStartSLOduration=1.568212154 podStartE2EDuration="1.992594216s" totalImagesPullingTime="424.382062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:45.991332918 +0000 UTC m=+4989.441438965" watchObservedRunningTime="2026-11-03 21:24:45.992594216 +0000 UTC m=+4989.442700261" Nov 03 21:24:46.687315 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:46.687278 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382ef016-7441-40f0-8151-0ebab3e5bd30" path="/var/lib/kubelet/pods/382ef016-7441-40f0-8151-0ebab3e5bd30/volumes" Nov 03 21:24:47.639999 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:24:47.639953 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:24:49.028474 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:49.027491 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz"] Nov 03 21:24:49.042464 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:49.040427 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-h2vxz"] Nov 03 21:24:50.679385 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:50.679339 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6" path="/var/lib/kubelet/pods/f0f4a1f9-d15d-4b20-8d5a-af3cc10160d6/volumes" Nov 03 21:24:52.013666 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:24:52.013634 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b4d8b8448-4m7sb" Nov 03 21:25:02.484832 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:25:02.484770 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:02.616481 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:25:02.616431 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:17.531198 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:25:17.531145 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:25:29.150397 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:25:29.150324 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:25:40.929486 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:25:40.929436 4711 scope.go:122] "RemoveContainer" containerID="f39267bb0cce8a7b68faf28d7ca9eb36daf8692ba749a2276d6ddce9889b1841" Nov 03 21:25:40.938171 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:25:40.938146 4711 scope.go:122] "RemoveContainer" containerID="7ecc696ecc5f4482b2ba8102304aa735b3fe757330e12b35b5f94c7d48a96fc6" Nov 03 21:25:40.945939 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:25:40.945923 4711 scope.go:122] "RemoveContainer" containerID="e583be9166122802cf830e94b3a02fb801ee3f1e1ddd2dabf5c671bd805818c0" Nov 03 21:25:47.638522 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:25:47.633767 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:26:01.076620 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:01.076580 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:02.441185 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:02.441144 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:06.632741 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:06.632695 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:08.529922 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:08.529874 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:08.667576 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:08.667541 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:13.026384 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:13.026340 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9"] Nov 03 21:26:13.028394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:13.028367 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4097800-wwxh9"] Nov 03 21:26:14.678878 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:14.678844 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864620b6-e214-49f5-908c-106c3b03bab9" path="/var/lib/kubelet/pods/864620b6-e214-49f5-908c-106c3b03bab9/volumes" Nov 03 21:26:17.533436 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:26:17.533398 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:26:19.531868 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:19.531831 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:19.662582 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:19.662545 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:26:30.734612 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:30.734568 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:26:30.735567 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:30.735524 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:26:38.035347 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:38.035320 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:26:38.038317 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:38.038298 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:26:38.070831 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:38.070811 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-56d9g_a3f23518-97e0-4f3f-b939-d76052dc1339/kube-multus/0.log" Nov 03 21:26:38.073512 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:38.073494 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-txw5p_9af6745a-681c-44b6-861c-310df033cfe9/ovn-acl-logging/0.log" Nov 03 21:26:40.977532 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:40.977498 4711 scope.go:122] "RemoveContainer" containerID="fe79f3d04440182b4e20f788d3eac571eedeff3fc59a5ee7afdfb0f806718683" Nov 03 21:26:40.986308 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:26:40.986286 4711 scope.go:122] "RemoveContainer" containerID="b1d8658395c2606870ff7e1c8da5b8fc17b607ac7978ec61d547ff89d8fb8b9d" Nov 03 21:26:47.545075 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:26:47.545030 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:27:01.250567 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:01.250527 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:02.564852 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:02.564772 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:15.737726 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:15.737678 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-7wgtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:27:15.738548 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:15.738493 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-7wgtr" podUID="c1894d83-96c7-4868-b7bb-0ce763ad9893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:27:16.141930 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:16.141887 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:16.269221 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:16.269182 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:17.532668 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:27:17.532630 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:27:24.032390 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:24.032324 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76"] Nov 03 21:27:24.034684 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:24.034658 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s6c76"] Nov 03 21:27:24.678470 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:24.678428 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16acce55-4861-46fc-84a3-6035346e9cf9" path="/var/lib/kubelet/pods/16acce55-4861-46fc-84a3-6035346e9cf9/volumes" Nov 03 21:27:37.205468 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:37.202897 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:27:41.010055 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:27:41.010024 4711 scope.go:122] "RemoveContainer" containerID="ceb078245b07d3ab29e52faff5e6181f5d46ab6b7b7b59754508d71fc79c8817" Nov 03 21:27:47.530933 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:27:47.530896 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:28:01.308415 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:28:01.308369 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:02.533092 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:28:02.533045 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:28:17.645428 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:28:17.645385 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:28:38.818682 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:28:38.818638 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:28:38.819325 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:28:38.819287 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:28:41.030467 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:28:41.030427 4711 scope.go:122] "RemoveContainer" containerID="7fc49fbe9d99158b86a3232f39bf10c1d7f9a02acfcfceafca12bfa52e3f698b" Nov 03 21:28:41.039063 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:28:41.039034 4711 scope.go:122] "RemoveContainer" containerID="b7ddfed32fc037accc3e4bf91af1680f41a75c431134416aa0a6a75750660597" Nov 03 21:28:47.675535 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:28:47.674633 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:29:01.386509 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:01.386470 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:02.587042 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:02.587004 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:29:08.801664 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:08.801622 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-jbqn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:29:08.802328 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:08.802292 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-jbqn5" podUID="a732ad93-52e7-436f-94b7-03ddd72075da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:29:08.828562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:08.828530 4711 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-59f957fdd7-qq2td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:29:08.828874 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:08.828842 4711 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-59f957fdd7-qq2td" podUID="d56f20a7-a8f7-4aed-86c2-28929da807ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:29:11.022771 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:11.022730 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt"] Nov 03 21:29:11.026269 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:11.026238 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gdgjt"] Nov 03 21:29:12.678704 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:12.678670 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6588d280-e10c-4457-8fd6-50c2385d809e" path="/var/lib/kubelet/pods/6588d280-e10c-4457-8fd6-50c2385d809e/volumes" Nov 03 21:29:17.622470 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:29:17.622299 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:29:29.022038 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:29.021984 4711 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb"] Nov 03 21:29:29.025253 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:29.025220 4711 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-r27lb"] Nov 03 21:29:30.679708 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:30.679674 4711 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76283c15-c9bc-4d94-a136-e8a36d2a225b" path="/var/lib/kubelet/pods/76283c15-c9bc-4d94-a136-e8a36d2a225b/volumes" Nov 03 21:29:47.530991 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:29:47.530951 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:29:48.108562 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:48.108527 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-69ff9b4cb6-z65tg_81a76879-bf5a-4088-a863-5e5ec0b7213b/manager/0.log" Nov 03 21:29:48.489466 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:48.489376 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ncvvt_de0a63b3-de1f-4ead-9e75-44a17a440fc8/manager/0.log" Nov 03 21:29:50.483217 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:50.483185 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b4d8b8448-4m7sb_60d63895-e621-40e6-8cd3-0c43df29016f/maas-api/0.log" Nov 03 21:29:58.640911 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:58.640879 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594f764c95-g9xdm_9fc557b1-89ba-4c1b-ae42-90b1806afcf8/kube-auth-proxy/0.log" Nov 03 21:29:59.192553 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:29:59.192518 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-7wgtr_c1894d83-96c7-4868-b7bb-0ce763ad9893/istio-proxy/0.log" Nov 03 21:30:00.165135 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:00.165095 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-jbqn5_a732ad93-52e7-436f-94b7-03ddd72075da/istio-proxy/0.log" Nov 03 21:30:00.356940 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:00.356911 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-qq2td_d56f20a7-a8f7-4aed-86c2-28929da807ee/istio-proxy/0.log" Nov 03 21:30:01.407030 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:01.406982 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:02.602851 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:02.602823 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-8d2rc_8de1d23d-be7a-4eaa-a606-bee092c35e74/main/0.log" Nov 03 21:30:02.642156 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:02.642126 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:02.778251 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:02.778223 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-lr7kb_669af6be-7305-4087-aafd-16fd9d48a4be/main/0.log" Nov 03 21:30:03.126770 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:03.126738 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8rqcl_e6c8976a-839f-442c-9c08-a0f53663f5d4/main/0.log" Nov 03 21:30:03.482829 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:03.482748 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-cjklg_6ee62e63-1708-4a53-8f0d-cb6875a87267/main/0.log" Nov 03 21:30:03.845248 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:03.845214 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-tdxw5_d59d95c7-b4a1-4307-8227-09e1a43822cb/main/0.log" Nov 03 21:30:13.792537 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:13.792505 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:13.927394 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:13.927359 4711 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" limit=1024 Nov 03 21:30:17.211806 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:17.211717 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bhl7x_a9998949-10c8-4511-857c-dfad4001ab59/global-pull-secret-syncer/0.log" Nov 03 21:30:17.457930 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:17.457895 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8fpvs_042f2f33-8a62-4464-bdd8-1a54ed18272d/konnectivity-agent/0.log" Nov 03 21:30:17.531177 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:30:17.531139 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:30:17.622505 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:17.622475 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-5.us-east-2.compute.internal_23259ea2648a683e23b5758ee6ae68e7/haproxy/0.log" Nov 03 21:30:27.174818 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.174780 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdbf29b0-1511-48dd-b837-fbf41e7a47a4/alertmanager/0.log" Nov 03 21:30:27.203000 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.202973 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdbf29b0-1511-48dd-b837-fbf41e7a47a4/config-reloader/0.log" Nov 03 21:30:27.228682 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.228658 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdbf29b0-1511-48dd-b837-fbf41e7a47a4/kube-rbac-proxy-web/0.log" Nov 03 21:30:27.258651 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.258630 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdbf29b0-1511-48dd-b837-fbf41e7a47a4/kube-rbac-proxy/0.log" Nov 03 21:30:27.307841 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.307811 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdbf29b0-1511-48dd-b837-fbf41e7a47a4/kube-rbac-proxy-metric/0.log" Nov 03 21:30:27.338581 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.338554 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdbf29b0-1511-48dd-b837-fbf41e7a47a4/prom-label-proxy/0.log" Nov 03 21:30:27.363274 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.363250 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cdbf29b0-1511-48dd-b837-fbf41e7a47a4/init-config-reloader/0.log" Nov 03 21:30:27.565202 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.565168 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5bc95fdfb7-hjwb4_30310e93-8e08-41c0-9faf-b0be1bef5df7/metrics-server/0.log" Nov 03 21:30:27.594865 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.594841 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-72rf7_48141310-02b0-418f-8ee6-70cf94ca1a19/monitoring-plugin/0.log" Nov 03 21:30:27.722234 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.722200 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2skn_90ef7a29-08bf-4a3c-b815-a34698896df1/node-exporter/0.log" Nov 03 21:30:27.747991 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.747969 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2skn_90ef7a29-08bf-4a3c-b815-a34698896df1/kube-rbac-proxy/0.log" Nov 03 21:30:27.772328 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.772310 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2skn_90ef7a29-08bf-4a3c-b815-a34698896df1/init-textfile/0.log" Nov 03 21:30:27.995518 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:27.995414 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_665e6a47-3a75-4135-a005-6c2f432e7495/prometheus/0.log" Nov 03 21:30:28.020110 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.020085 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_665e6a47-3a75-4135-a005-6c2f432e7495/config-reloader/0.log" Nov 03 21:30:28.045435 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.045409 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_665e6a47-3a75-4135-a005-6c2f432e7495/thanos-sidecar/0.log" Nov 03 21:30:28.071727 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.071703 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_665e6a47-3a75-4135-a005-6c2f432e7495/kube-rbac-proxy-web/0.log" Nov 03 21:30:28.096918 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.096890 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_665e6a47-3a75-4135-a005-6c2f432e7495/kube-rbac-proxy/0.log" Nov 03 21:30:28.128545 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.128521 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_665e6a47-3a75-4135-a005-6c2f432e7495/kube-rbac-proxy-thanos/0.log" Nov 03 21:30:28.156693 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.156670 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_665e6a47-3a75-4135-a005-6c2f432e7495/init-config-reloader/0.log" Nov 03 21:30:28.199035 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.199014 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-zb6c5_2ffacfcf-65b1-4809-8930-3c2952a90fc8/prometheus-operator/0.log" Nov 03 21:30:28.220030 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.220003 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-zb6c5_2ffacfcf-65b1-4809-8930-3c2952a90fc8/kube-rbac-proxy/0.log" Nov 03 21:30:28.253006 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.252944 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-hh4rb_cd19fe74-5279-4b82-90e2-37bee00f7ccc/prometheus-operator-admission-webhook/0.log" Nov 03 21:30:28.287638 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.287614 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-687d6677d-h77qd_fe593514-53cf-4c0b-a669-f853d064502b/telemeter-client/0.log" Nov 03 21:30:28.320347 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.320319 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-687d6677d-h77qd_fe593514-53cf-4c0b-a669-f853d064502b/reload/0.log" Nov 03 21:30:28.347813 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.347793 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-687d6677d-h77qd_fe593514-53cf-4c0b-a669-f853d064502b/kube-rbac-proxy/0.log" Nov 03 21:30:28.381947 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.381905 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654645b88f-f8qq9_4d10424a-4e8c-4d2f-bbf9-f9b165f1685e/thanos-query/0.log" Nov 03 21:30:28.412534 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.412509 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654645b88f-f8qq9_4d10424a-4e8c-4d2f-bbf9-f9b165f1685e/kube-rbac-proxy-web/0.log" Nov 03 21:30:28.446068 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.446046 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654645b88f-f8qq9_4d10424a-4e8c-4d2f-bbf9-f9b165f1685e/kube-rbac-proxy/0.log" Nov 03 21:30:28.471972 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.471951 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654645b88f-f8qq9_4d10424a-4e8c-4d2f-bbf9-f9b165f1685e/prom-label-proxy/0.log" Nov 03 21:30:28.496724 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.496701 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654645b88f-f8qq9_4d10424a-4e8c-4d2f-bbf9-f9b165f1685e/kube-rbac-proxy-rules/0.log" Nov 03 21:30:28.520749 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.520716 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-654645b88f-f8qq9_4d10424a-4e8c-4d2f-bbf9-f9b165f1685e/kube-rbac-proxy-metrics/0.log" Nov 03 21:30:28.708428 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.708397 4711 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn"] Nov 03 21:30:28.709072 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.709041 4711 cpu_manager.go:414] "containerMap: removing container" podUID="382ef016-7441-40f0-8151-0ebab3e5bd30" containerName="maas-api" Nov 03 21:30:28.709072 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.709066 4711 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382ef016-7441-40f0-8151-0ebab3e5bd30" containerName="maas-api" Nov 03 21:30:28.709255 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.709159 4711 memory_manager.go:365] "RemoveStaleState removing state" podUID="382ef016-7441-40f0-8151-0ebab3e5bd30" containerName="maas-api" Nov 03 21:30:28.712560 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.712540 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.727376 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.727350 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn"] Nov 03 21:30:28.764167 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.764137 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-podres\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.764313 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.764193 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-sys\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.764313 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.764262 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-lib-modules\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.764467 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.764321 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6fs\" (UniqueName: \"kubernetes.io/projected/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-kube-api-access-nt6fs\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.764467 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.764364 4711 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-proc\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.810789 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.810714 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjmcb\"/\"kube-root-ca.crt\"" Nov 03 21:30:28.810926 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.810828 4711 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fjmcb\"/\"openshift-service-ca.crt\"" Nov 03 21:30:28.810983 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.810947 4711 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fjmcb\"/\"default-dockercfg-sl6jl\"" Nov 03 21:30:28.865395 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.865372 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-sys\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.865544 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.865404 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-lib-modules\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.865544 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.865424 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6fs\" (UniqueName: \"kubernetes.io/projected/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-kube-api-access-nt6fs\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.865544 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.865504 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-sys\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.865734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.865545 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-lib-modules\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.865734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.865569 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-proc\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.865734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.865615 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-proc\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.865734 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.865697 4711 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-podres\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.865907 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.865788 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-podres\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:28.877938 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:28.877910 4711 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6fs\" (UniqueName: \"kubernetes.io/projected/ef802f6c-09e3-46e8-9aeb-729a187c8e5e-kube-api-access-nt6fs\") pod \"perf-node-gather-daemonset-mtvmn\" (UID: \"ef802f6c-09e3-46e8-9aeb-729a187c8e5e\") " pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:29.025665 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:29.025634 4711 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:29.301140 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:29.301113 4711 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn"] Nov 03 21:30:29.303516 ip-10-0-142-5 kubenswrapper[4711]: W1103 21:30:29.303487 4711 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podef802f6c_09e3_46e8_9aeb_729a187c8e5e.slice/crio-8a4a40f7b84ae21bcfb97ced245ededff22e032c45217e1f0ce546562f54099a WatchSource:0}: Error finding container 8a4a40f7b84ae21bcfb97ced245ededff22e032c45217e1f0ce546562f54099a: Status 404 returned error can't find the container with id 8a4a40f7b84ae21bcfb97ced245ededff22e032c45217e1f0ce546562f54099a Nov 03 21:30:29.433156 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:29.433128 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" event={"ID":"ef802f6c-09e3-46e8-9aeb-729a187c8e5e","Type":"ContainerStarted","Data":"f6563081f059dafe7347afc13fda5c8ff52e0d35b6ead6e7a25eb5c1cfc82cfd"} Nov 03 21:30:29.433284 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:29.433174 4711 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" event={"ID":"ef802f6c-09e3-46e8-9aeb-729a187c8e5e","Type":"ContainerStarted","Data":"8a4a40f7b84ae21bcfb97ced245ededff22e032c45217e1f0ce546562f54099a"} Nov 03 21:30:29.433284 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:29.433274 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:29.454122 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:29.454081 4711 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" podStartSLOduration=1.454067156 podStartE2EDuration="1.454067156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:29.451990218 +0000 UTC m=+5332.902096264" watchObservedRunningTime="2026-11-03 21:30:29.454067156 +0000 UTC m=+5332.904173239" Nov 03 21:30:31.558146 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:31.558113 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-smhql_1e17c916-ecd2-4ccd-bfeb-451380f47d30/download-server/0.log" Nov 03 21:30:33.699170 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:33.699131 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c8hzq_7e75b735-0229-429d-927f-5683bcd373d6/dns/0.log" Nov 03 21:30:33.719903 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:33.719873 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c8hzq_7e75b735-0229-429d-927f-5683bcd373d6/kube-rbac-proxy/0.log" Nov 03 21:30:33.925352 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:33.925328 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wszxg_fbdb17a3-3ead-4c90-9922-31fbae907537/dns-node-resolver/0.log" Nov 03 21:30:34.311985 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:34.311957 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-69ff9b4cb6-z65tg_81a76879-bf5a-4088-a863-5e5ec0b7213b/manager/0.log" Nov 03 21:30:34.457603 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:34.457568 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ncvvt_de0a63b3-de1f-4ead-9e75-44a17a440fc8/manager/0.log" Nov 03 21:30:34.641857 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:34.641780 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5h7vn_baba1acd-b516-4149-a337-d56b9e74b36e/node-ca/0.log" Nov 03 21:30:35.448704 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:35.448677 4711 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fjmcb/perf-node-gather-daemonset-mtvmn" Nov 03 21:30:39.468268 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:39.468232 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-69ff9b4cb6-z65tg_81a76879-bf5a-4088-a863-5e5ec0b7213b/manager/0.log" Nov 03 21:30:39.608702 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:39.608671 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ncvvt_de0a63b3-de1f-4ead-9e75-44a17a440fc8/manager/0.log" Nov 03 21:30:41.071199 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:41.071158 4711 scope.go:122] "RemoveContainer" containerID="b145d2c989562989e21d9b0f41ac3a336003fe2dbda881251cc7ec080eefc9f8" Nov 03 21:30:41.080011 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:41.079987 4711 scope.go:122] "RemoveContainer" containerID="fc1ffb15960452978c2f13c7433d64011100047b7fff6f565f321e0b1300d7f7" Nov 03 21:30:43.217104 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:43.217068 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594f764c95-g9xdm_9fc557b1-89ba-4c1b-ae42-90b1806afcf8/kube-auth-proxy/0.log" Nov 03 21:30:43.331071 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:43.331039 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-7wgtr_c1894d83-96c7-4868-b7bb-0ce763ad9893/istio-proxy/0.log" Nov 03 21:30:43.542751 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:43.542712 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-jbqn5_a732ad93-52e7-436f-94b7-03ddd72075da/istio-proxy/0.log" Nov 03 21:30:43.576934 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:43.576905 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-59f957fdd7-qq2td_d56f20a7-a8f7-4aed-86c2-28929da807ee/istio-proxy/0.log" Nov 03 21:30:45.321664 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:45.321631 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-69ff9b4cb6-z65tg_81a76879-bf5a-4088-a863-5e5ec0b7213b/manager/0.log" Nov 03 21:30:45.461616 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:45.461586 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ncvvt_de0a63b3-de1f-4ead-9e75-44a17a440fc8/manager/0.log" Nov 03 21:30:46.610213 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:46.610177 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7wjnx_91a2c232-0afd-43ed-a457-c977e4961416/serve-healthcheck-canary/0.log" Nov 03 21:30:47.532337 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:47.532303 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-znlzm_92b5617d-95cb-44c2-84b5-a73a50c2d749/kube-rbac-proxy/0.log" Nov 03 21:30:47.533931 ip-10-0-142-5 kubenswrapper[4711]: E1103 21:30:47.533898 4711 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-znlzm" containerName="extractor" Nov 03 21:30:47.558226 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:47.558197 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-znlzm_92b5617d-95cb-44c2-84b5-a73a50c2d749/exporter/0.log" Nov 03 21:30:47.591779 ip-10-0-142-5 kubenswrapper[4711]: I1103 21:30:47.591752 4711 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-znlzm_92b5617d-95cb-44c2-84b5-a73a50c2d749/extractor/0.log"